SDLC Corp Software Development

Progressive Image Loading From Scratch

Samcorp — Thu, 03 Sep 2026 10:39:51 GMT

Large images can make a page feel slow even when the rest of the interface loads quickly.

Progressive image loading improves perceived performance by showing a lightweight placeholder first and replacing it with the full-quality image once it has downloaded and decoded.

The typical flow looks like this:

Tiny Placeholder
      ↓
Full Image Downloads
      ↓
Image Decodes
      ↓
Smooth Fade-In

In this tutorial, we will build that behavior using plain HTML, CSS, and JavaScript.

What Is Progressive Image Loading?

Progressive image loading does not necessarily make the final image file smaller. Instead, it improves the loading experience.

Without progressive loading:

Blank Area
    ↓
Wait
    ↓
Large Image Appears

With progressive loading:

Small Blurred Preview
        ↓
Full Image Loads
        ↓
Sharp Image Replaces Preview

This approach is sometimes called the blur-up technique.

It works especially well for:

Blog images
Product galleries
Portfolio pages
Image-heavy landing pages
Ecommerce cards

Step 1: Create the HTML

Start with a wrapper containing a low-resolution placeholder and the full image.

The first image should be extremely small.

For example:

mountain-small.jpg → 20–50 px wide
mountain-large.jpg → 1200 px wide

The tiny version downloads almost immediately.

The explicit width and height attributes are important because they reserve space before the image finishes loading and help prevent layout shifts.

Step 2: Layer the Images With CSS

Now place both images in the same area.

.progressive-image {
  position: relative;
  overflow: hidden;
  aspect-ratio: 3 / 2;
  background: #eee;
}

.progressive-image img {
  width: 100%;
  height: 100%;
  object-fit: cover;
}

.placeholder {
  position: absolute;
  inset: 0;
  filter: blur(18px);
  transform: scale(1.08);
}

.full-image {
  position: absolute;
  inset: 0;
  opacity: 0;
  transition: opacity 350ms ease;
}

The placeholder is intentionally blurred.

The slight scaling:

transform: scale(1.08);

helps hide blurry edges created by the CSS filter.

Step 3: Reveal the Full Image When It Loads

Add a class after the high-resolution image is ready.

document
  .querySelectorAll(".progressive-image")
  .forEach((wrapper) => {
    const image = wrapper.querySelector(".full-image");

    image.addEventListener("load", () => {
      image.classList.add("loaded");
    });
  });

Then add the following CSS:

.full-image.loaded {
  opacity: 1;
}

Now the loading sequence becomes:

Blurred Preview
      ↓
Large Image Loads
      ↓
opacity: 0 → 1
      ↓
Sharp Image

Step 4: Wait for Image Decoding

The network request finishing does not always mean the browser has finished decoding the image.

For smoother replacement, use:

image.decode()

The decode() method returns a promise that resolves after the image has been decoded and is ready to render.

Update the JavaScript:

document
  .querySelectorAll(".progressive-image")
  .forEach(async (wrapper) => {
    const image = wrapper.querySelector(".full-image");

    try {
      if (!image.complete) {
        await new Promise((resolve, reject) => {
          image.addEventListener(
            "load",
            resolve,
            { once: true }
          );

          image.addEventListener(
            "error",
            reject,
            { once: true }
          );
        });
      }

      await image.decode();

      image.classList.add("loaded");
    } catch {
      wrapper.classList.add("image-error");
    }
  });

This avoids fading in an image before it is actually ready to paint.

Step 5: Lazy Load Images Below the Fold

For images that are not visible immediately, let the browser delay the full-resolution request.

Add:

Native loading="lazy" tells the browser that off-screen images can be deferred until they approach the viewport.

Use it primarily for images below the fold.

Do Not Lazy Load Your Hero Image

If an image is visible immediately and may become your Largest Contentful Paint element, lazy loading can delay it unnecessarily.

For a hero image, use:

fetchpriority="high" tells the browser that the image is relatively important. This hint should be used sparingly because assigning high priority to too many resources can reduce its effectiveness.

A simple rule is:

Hero / LCP Image
→ eager + fetchpriority="high"

Below-the-fold Images
→ loading="lazy"

Step 6: Add Responsive Images

Progressive loading should not mean sending a 2000-pixel image to every device.

Use srcset:

The browser can now choose the most appropriate image based on the display size and device pixel ratio.

Using srcset and sizes helps prevent browsers from downloading unnecessarily large images on smaller devices.

Responsive image handling is also an important part of performance-focused web development, especially when teams are optimizing Core Web Vitals, mobile layouts, and page-load speed.

Step 7: Remove the Placeholder After Loading

Once the full image is visible, the placeholder is no longer necessary.

Update the code:

await image.decode();

image.classList.add("loaded");

image.addEventListener(
  "transitionend",
  () => {
    const placeholder =
      wrapper.querySelector(".placeholder");

    placeholder?.remove();
  },
  { once: true }
);

Now the browser can discard the unnecessary placeholder element after the transition finishes.

Complete Progressive Image Loading Example

HTML

CSS

.progressive-image {
  position: relative;
  overflow: hidden;
  aspect-ratio: 3 / 2;
  background: #eee;
}

.progressive-image img {
  width: 100%;
  height: 100%;
  object-fit: cover;
}

.placeholder {
  position: absolute;
  inset: 0;
  filter: blur(18px);
  transform: scale(1.08);
}

.full-image {
  position: absolute;
  inset: 0;
  opacity: 0;
  transition: opacity 350ms ease;
}

.full-image.loaded {
  opacity: 1;
}

JavaScript

document
  .querySelectorAll(".progressive-image")
  .forEach(async (wrapper) => {
    const image = wrapper.querySelector(".full-image");

    try {
      if (!image.complete) {
        await new Promise((resolve, reject) => {
          image.addEventListener(
            "load",
            resolve,
            { once: true }
          );

          image.addEventListener(
            "error",
            reject,
            { once: true }
          );
        });
      }

      await image.decode();

      image.classList.add("loaded");

      image.addEventListener(
        "transitionend",
        () => {
          wrapper
            .querySelector(".placeholder")
            ?.remove();
        },
        { once: true }
      );
    } catch {
      wrapper.classList.add("image-error");
    }
  });

What About JavaScript-Based Lazy Loading?

Older implementations often used IntersectionObserver to decide when to assign an image URL.

That approach still has valid use cases when you need custom loading behavior, but native:

loading="lazy"

is usually simpler for standard image lazy loading.

Intersection Observer remains useful when you need more control over exactly when elements enter or leave the viewport.

Avoid adding custom JavaScript when the browser already provides the behavior you need.

Progressive Loading vs Progressive JPEG

These two concepts are related but different.

Progressive JPEG encodes one image so it appears in multiple visual passes while downloading.

The technique used in this tutorial relies on:

Tiny Separate Image
        +
Full Resolution Image
        +
CSS Transition

This gives developers more control over:

Placeholders
Responsive images
Lazy loading
Image decoding
Transition behavior

Common Progressive Image Loading Mistakes

Avoid these common issues:

Lazy loading the main hero or LCP image
Serving the same huge image to every device
Omitting width and height
Using a placeholder that is still hundreds of kilobytes
Applying excessive blur effects
Loading the full image twice
Waiting only for load when smooth decoding matters
Forgetting meaningful alt text
Writing complex JavaScript when native lazy loading is enough

The goal is not to make the implementation complicated.

The goal is to improve perceived loading speed without harming actual performance.

A strong frontend user experience strategy should balance visual quality with performance so placeholders, responsive layouts, and loading states improve the experience instead of distracting users.

Recommended Image Loading Strategy

A practical setup looks like this:

Above-the-Fold Images

Above the Fold
      ↓
Responsive Image
      ↓
fetchpriority="high"
      ↓
Display Immediately

Below-the-Fold Images

Below the Fold
      ↓
Tiny Placeholder
      ↓
loading="lazy"
      ↓
Responsive Full Image
      ↓
decode()
      ↓
Fade In

This approach prioritizes important visual content while delaying resources that users do not immediately need.

Final Thoughts

Good progressive image loading combines several small techniques rather than relying on a single trick.

Use a lightweight placeholder to give users immediate visual feedback. Reserve image dimensions to prevent layout shifts. Let native lazy loading defer non-critical images, use responsive sources to avoid unnecessary downloads, and wait for decoding before revealing the final image.

The simplest rule is:

Load important images early.
Defer unimportant images.
Never download more image data than necessary.

Following these principles gives users a smoother image-loading experience without introducing a heavy image-loading library.

Structured Output Parsing That Doesn't Break

Samcorp — Wed, 02 Sep 2026 09:17:33 GMT

Getting an LLM to return JSON is easy, Getting it to return JSON that your application can safely depend on is harder.

A production LLM structured output pipeline needs more than a prompt saying:

Return valid JSON.

You need a schema, validation, error handling, retry rules, and a clear strategy for incomplete responses and refusals.

In this tutorial, we'll build a safer structured-output flow using Python, Pydantic, and schema-constrained model output.

Why Plain JSON Prompts Break

A common approach looks like this:

prompt = """
Extract the support ticket.

Return JSON with:
- category
- priority
- summary
"""

The response might be:

{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice."
}

Looks fine.

But another request might return:

Here is the JSON you requested:

{
  "category": "billing",
  "priority": "urgent",
  "summary": "Customer was charged twice."
}

Now you have several problems:

Extra prose before the JSON
An unexpected value such as urgent
Missing fields
Wrong data types
Partial JSON after token truncation
Fields your application never expected

The fix is not a more aggressive regex.

The fix is to treat model output like any other untrusted external input.

Step 1: Define the Contract First

Start with the shape your application needs.

Using Pydantic:

from typing import Literal
from pydantic import BaseModel


class SupportTicket(BaseModel):
    category: Literal[
        "billing",
        "account",
        "technical",
        "other"
    ]

    priority: Literal[
        "low",
        "medium",
        "high"
    ]

    summary: str
    customer_name: str | None

Now your application has a clear contract.

Valid

{
  "category": "billing",
  "priority": "high",
  "summary": "Customer was charged twice.",
  "customer_name": "Alex"
}

Invalid

{
  "category": "payments",
  "priority": "urgent"
}

The schema is now the source of truth—not the prompt.

Step 2: Use Schema-Constrained Output

Install the SDKs:

pip install openai pydantic

Then ask the model to return the Pydantic structure directly:

from openai import OpenAI

client = OpenAI()

response = client.responses.parse(
    model="gpt-5.6",
    input=[
        {
            "role": "system",
            "content": (
                "Extract the support ticket "
                "information from the message."
            ),
        },
        {
            "role": "user",
            "content": (
                "Alex says their card was charged "
                "twice for the same subscription."
            ),
        },
    ],
    text_format=SupportTicket,
)

ticket = response.output_parsed

print(ticket)

The current OpenAI SDK can parse Structured Outputs directly into Pydantic models, reducing the need for manual json.loads() plumbing.

Conceptually:

User Input
    ↓
LLM
    ↓
JSON Schema
    ↓
Structured Output
    ↓
Typed Application Object

Step 3: Do Not Confuse JSON With Valid Data

These are different guarantees:

Valid JSON
≠
Correct Schema
≠
Correct Business Data

For example:

{
  "category": "billing",
  "priority": "high",
  "summary": "Refund requested.",
  "customer_name": null
}

This may perfectly match the schema.

But perhaps your system requires a customer ID before a refund workflow can start.

Schema validation cannot know every business rule.

Use another validation layer:

def validate_ticket(ticket: SupportTicket):
    if (
        ticket.category == "billing"
        and not ticket.summary.strip()
    ):
        raise ValueError(
            "Billing tickets require a summary."
        )

Think in three layers:

Syntax validation
      ↓
Schema validation
      ↓
Business validation

Teams building production-grade LLM workflows can also explore custom generative AI development for output validation, guardrails, model integration, and reliable AI application delivery.

Step 4: Handle Nullable Fields Explicitly

Do not ask the model to invent missing information.

If a customer name may not exist, model that explicitly:

customer_name: str | None

Then:

{
  "customer_name": null
}

is better than:

{
  "customer_name": "Unknown User"
}

unless "Unknown User" has a real meaning in your system.

For current OpenAI Structured Outputs, schema fields are required; optional behavior can be represented using a nullable type.

That creates a useful rule:

Missing information should be represented as missing—not guessed.

Step 5: Handle Refusals Separately

Even when you request structured output, the model may refuse some user inputs.

A refusal should not be treated as malformed JSON.

Current OpenAI responses expose refusals separately because a refusal does not necessarily follow your requested schema.

A simple helper:

def find_refusal(response):
    for output in response.output:
        if output.type != "message":
            continue

        for item in output.content:
            if item.type == "refusal":
                return item.refusal

    return None

Then:

refusal = find_refusal(response)

if refusal:
    print("Request refused:", refusal)
else:
    ticket = response.output_parsed

Do not automatically retry a refusal as though it were a parsing error.

Step 6: Detect Incomplete Responses

Structured output can still fail if generation is interrupted.

For example:

Token limit reached
        ↓
Response stops early
        ↓
Structured object incomplete

OpenAI's documentation explicitly recommends checking for incomplete responses, including cases where the maximum output-token limit was reached.

Use a guard:

if response.status == "incomplete":
    reason = response.incomplete_details.reason

    raise RuntimeError(
        f"Incomplete model response: {reason}"
    )

Never send a half-generated object deeper into your application.

Step 7: Fail Closed When Parsing Fails

Avoid this:

ticket = response.output_parsed or {}

It hides the failure.

Later your code may do:

ticket["priority"]

and fail somewhere completely unrelated.

Instead:

ticket = response.output_parsed

if ticket is None:
    raise ValueError(
        "Model did not return valid structured data."
    )

Failing close to the source makes debugging much easier.

Step 8: Add Controlled Retries

Some failures can be retried.

A simple approach:

def parse_ticket(message, attempts=2):
    last_error = None

    for _ in range(attempts):
        try:
            response = client.responses.parse(
                model="gpt-5.6",
                input=[
                    {
                        "role": "system",
                        "content":
                            "Extract support ticket data.",
                    },
                    {
                        "role": "user",
                        "content": message,
                    },
                ],
                text_format=SupportTicket,
            )

            if response.status == "incomplete":
                raise RuntimeError(
                    "Incomplete response"
                )

            refusal = find_refusal(response)

            if refusal:
                raise PermissionError(refusal)

            if response.output_parsed is None:
                raise ValueError(
                    "Structured parsing failed"
                )

            return response.output_parsed

        except PermissionError:
            raise

        except Exception as exc:
            last_error = exc

    raise RuntimeError(
        "Unable to obtain valid structured output"
    ) from last_error

The important part is not the exact retry count.

It is deciding which failures deserve a retry.

Do not endlessly retry:

Invalid input
Refusals
Business-rule failures
Permanent configuration errors

Retries should be bounded.

Step 9: Avoid Regex-Based JSON Extraction

This pattern is fragile:

import re

match = re.search(r"\{.*\}", output, re.DOTALL)

It may break with:

Nested objects
Braces inside strings
Multiple JSON objects
Markdown
Truncated responses

If your provider supports structured schema output, use it.

If it supports only JSON mode, parse JSON normally and then validate it.

For example:

import json

raw = json.loads(model_output)

ticket = SupportTicket.model_validate(raw)

Or validate JSON directly:

ticket = SupportTicket.model_validate_json(
    model_output
)

Pydantic provides built-in JSON parsing and validation through model_validate_json().

Step 10: Keep Schemas Small

Do not start with a giant structure containing 50 fields.

Instead of:

Customer
 ├── Profile
 ├── Addresses
 ├── Orders
 ├── Refunds
 ├── Preferences
 ├── Marketing
 └── Support History

extract only what the current operation requires.

For example:

class RefundIntent(BaseModel):
    order_id: str | None
    reason: str
    requested: bool

Smaller schemas are easier to:

Understand
Test
Version
Validate
Monitor
Change safely

Step 11: Version Your Output Contract

Eventually your schema will change.

Version 1

{
  "category": "billing",
  "summary": "Duplicate payment"
}

Version 2

Version 2 might add:

{
  "category": "billing",
  "summary": "Duplicate payment",
  "priority": "high"
}

Treat this like an API change.

A practical pattern is:

support_ticket_v1
support_ticket_v2

or include an explicit application-side schema version.

Do not silently change a structure that other services already depend on.

Step 12: Test With Bad Inputs

Do not test only perfect prompts.

Try cases like:

Empty input
Very long input
Missing information
Conflicting information
Multiple entities
Unexpected languages
Prompt injection attempts
Ambiguous dates
Malformed IDs

Then verify:

Did the schema hold?
Did missing values become null?
Was a refusal handled?
Did business validation catch problems?
Did retries stop correctly?

Structured output should be tested like an API boundary.

For applications that process large volumes of unstructured text, NLP-powered automation solutions can help structure, validate, and integrate model-generated data into downstream workflows.

A Production-Safe Parsing Flow

A reliable LLM structured output pipeline looks like this:

User Input
    ↓
Schema-Constrained Generation
    ↓
Response Complete?
   / \
 No   Yes
 ↓     ↓
Fail  Refusal?
        / \
      Yes  No
       ↓    ↓
    Handle  Parse
              ↓
       Schema Validation
              ↓
       Business Validation
              ↓
         Trusted Object

Only the final object should enter the rest of your application.

Common Structured Output Mistakes

Avoid these patterns:

Asking for JSON only through prompt wording
Using regex to extract objects
Trusting valid JSON without schema validation
Allowing unknown enum values
Forcing the model to invent missing data
Ignoring truncated responses
Retrying refusals indefinitely
Swallowing parsing errors
Using enormous schemas
Changing output contracts without versioning

The goal is not simply to make the model output JSON.

The goal is to make the application behavior predictable when the model does something unexpected.

Final Thoughts

Reliable structured output comes from treating the LLM like an external service, not a trusted function.

Use a real schema. Validate the response. Represent missing information explicitly. Detect incomplete output and refusals. Add business validation after parsing, and retry only when the failure is actually recoverable.

A strong LLM structured output flow therefore looks like:

Generate
   ↓
Constrain
   ↓
Validate
   ↓
Handle Failure
   ↓
Use Data

Once you build that boundary correctly, downstream application code becomes much simpler because it no longer has to guess what shape the model decided to return.

Hybrid Search: Combining BM25 and Embeddings

Samcorp — Wed, 02 Sep 2026 08:56:19 GMT

Keyword search is excellent when users know the exact terms they are looking for. Semantic search is better when the query and the relevant document use different words.

In real applications, we usually need both.

Hybrid search with BM25 and embeddings combines lexical matching with semantic similarity so exact keywords, product codes, names, and technical terms remain important while conceptually related content can still rank.

In this tutorial, we'll build the core search flow and combine both result sets using Reciprocal Rank Fusion (RRF).

Why Combine BM25 and Embeddings?

Suppose our documents contain:

1. "How to reset your account password"
2. "Recover access when you cannot sign in"
3. "Configure OAuth authentication"

Now search for:

forgot login credentials

A keyword search may struggle because the document does not necessarily contain the exact words.

An embedding model can understand that:

forgot login credentials
        ≈
recover access when you cannot sign in

But semantic search has another weakness.

Consider:

ERR_CONNECTION_RESET

or:

SKU-48219

Exact lexical matching may be much more useful than semantic similarity.

That gives us:

Search Method	Good At
BM25	Keywords, names, IDs, exact terminology
Embeddings	Meaning, synonyms, natural-language queries
Hybrid Search	Combining both signals

OpenSearch uses BM25 as its default keyword-ranking algorithm and describes hybrid search as combining keyword and semantic retrieval to improve relevance.

The Hybrid Search Architecture

The basic flow looks like this:

                 User Query
                     |
             ┌───────┴───────┐
             ↓               ↓
        BM25 Search     Create Embedding
             ↓               ↓
      Keyword Results   Vector Search
             │               │
             └───────┬───────┘
                     ↓
                Rank Fusion
                     ↓
                Final Results

Instead of asking one retrieval method to solve every search problem, we let each produce its best candidates.

Then we combine them.

Step 1: Index Text for BM25

Let's use Elasticsearch for the example.

Create an index containing normal text fields:

PUT articles
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "content": {
        "type": "text"
      }
    }
  }
}

A normal match query uses lexical relevance:

GET articles/_search
{
  "query": {
    "match": {
      "content": "database connection timeout"
    }
  }
}

BM25 ranks documents based on signals such as term occurrence, document length, and how uncommon terms are across the collection.

This works especially well when exact terminology matters.

Step 2: Add Semantic Representations

For semantic search, documents need vector representations.

Conceptually:

"reset account password"
          ↓
     Embedding Model
          ↓
[0.14, -0.28, 0.67, ...]

The query goes through the same model:

"forgot my password"
          ↓
     Embedding Model
          ↓
[0.12, -0.25, 0.70, ...]

Vector search then finds nearby representations.

Sentence Transformers describes this process as embedding both the corpus and the query into the same vector space and retrieving entries with high semantic similarity. Cosine similarity is commonly used for this comparison.

Teams building search experiences around embeddings can also explore semantic search and NLP solutions for vector-based retrieval, contextual search, and recommendation systems.

In Elasticsearch, the current semantic_text workflow can manage embedding generation automatically.

A simplified mapping can contain both versions:

PUT knowledge-base
{
  "mappings": {
    "properties": {
      "content": {
        "type": "text"
      },
      "content_embedding": {
        "type": "semantic_text"
      }
    }
  }
}

The important design is:

content
→ lexical/BM25 search

content_embedding
→ semantic search

Elasticsearch currently recommends semantic_text as its simpler managed workflow for semantic retrieval.

Step 3: Run Both Searches

For a query such as:

fix slow database requests

Run lexical search against the regular field:

{
  "standard": {
    "query": {
      "match": {
        "content": "fix slow database requests"
      }
    }
  }
}

Then run semantic retrieval against the embedding-backed field:

{
  "standard": {
    "query": {
      "match": {
        "content_embedding": "fix slow database requests"
      }
    }
  }
}

The two searches may produce different rankings.

BM25 Results

1. Fixing slow database queries
2. Database request logging
3. Database timeout configuration

Semantic Search Results

1. Improving SQL query performance
2. Fixing slow database queries
3. Optimizing application data access

Both lists contain useful information.

We now need to merge them.

Step 4: Do Not Simply Add the Scores

A tempting approach is:

final_score =
    bm25_score +
    vector_similarity

That can be unreliable.

BM25 and vector similarity scores do not necessarily share the same scale. Elasticsearch specifically notes that BM25 scores can be unbounded while vector-related scores may use bounded ranges, making direct score combination difficult without normalization.

Instead, we can combine rank positions.

Step 5: Combine Results With RRF

Reciprocal Rank Fusion assigns a score based on where a document appears in each result list.

Conceptually:

RRF score =
1 / (k + BM25 rank)
+
1 / (k + vector rank)

For example:

Document A
BM25 rank:   1
Vector rank: 3

Document B
BM25 rank:   5
Vector rank: 1

A document that performs well in either or both systems can rise in the final ranking.

RRF is useful because it works on ranking positions rather than requiring BM25 and vector scores to be directly comparable.

Elasticsearch currently recommends RRF for hybrid retrieval.

Step 6: Create the Hybrid Query

With Elasticsearch's RRF retriever:

GET knowledge-base/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "match": {
                "content": "fix slow database requests"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "match": {
                "content_embedding": "fix slow database requests"
              }
            }
          }
        }
      ]
    }
  }
}

The first branch handles lexical matching.

The second handles semantic retrieval.

RRF merges both into one result list. Elasticsearch's current hybrid-search documentation uses this same pattern with separate text and semantic fields.

Step 7: Understand the Final Ranking

Suppose our searches return:

BM25 Results

1. Database request troubleshooting
2. Improving SQL query performance
3. Application latency debugging

Embedding Results

1. Improving SQL query performance
2. Optimizing slow backend systems
3. Database request troubleshooting

The hybrid ranking could become:

1. Improving SQL query performance
2. Database request troubleshooting
3. Optimizing slow backend systems
4. Application latency debugging

The first two documents receive strong signals from both retrieval methods.

That is the main advantage of hybrid retrieval.

Step 8: Add Metadata Filters

Most real search systems also need filters.

For example:

Search:
"authentication problems"

Filters:
category = documentation
language = English
status = published

Apply structured filters before or alongside retrieval so irrelevant documents do not become candidates simply because their text is similar.

Common filters include:

Tenant or organization ID
Product
Language
Region
Date
Permissions
Document type

For business applications, authorization filtering is especially important.

Search results should never expose content the current user cannot access.

Step 9: Tune Candidate Sizes

Vector search normally retrieves the nearest k documents.

Elasticsearch's kNN search also supports a larger candidate pool before selecting the final neighbors.

Approximate kNN is designed for scalable vector retrieval, while exact brute-force vector search becomes expensive on larger collections.

Conceptually:

Vector index
    ↓
Retrieve 100 candidates
    ↓
Keep best 20
    ↓
Combine with BM25 candidates
    ↓
RRF

Increasing the candidate pool can improve recall, but it also increases query work.

Benchmark it rather than choosing large values automatically.

Step 10: Evaluate Search Quality

Do not decide whether hybrid search works based on only a few manual searches.

Create a small evaluation set containing:

Query
Expected relevant documents

For example:

Query	Expected Result
`password reset`	Password recovery guide
`can't access account`	Password recovery guide
`ERR_CONNECTION_RESET`	Exact error troubleshooting
`speed up SQL`	Database optimization guide

Include multiple query types:

Exact keyword queries
+
Natural-language queries
+
Synonyms
+
Product IDs
+
Acronyms
+
Misspellings

Then compare:

BM25 only
vs.
Embeddings only
vs.
Hybrid

Useful search metrics include:

Precision@K
Recall@K
Mean Reciprocal Rank
NDCG

The best hybrid configuration depends on your own users and documents.

When Hybrid Search Works Best

Hybrid search with BM25 and embeddings is particularly useful for:

Documentation search
Enterprise knowledge bases
Ecommerce product discovery
RAG retrieval
Support-center search
Developer documentation
Internal company search

Hybrid retrieval is also useful in RAG and generative AI applications, where vector databases and retrieval pipelines help ground model responses in relevant enterprise data.

For example, a developer documentation system needs semantic understanding for:

"why is my API request being rejected?"

while still requiring precise matches for:

HTTP 429

Hybrid retrieval handles both cases better than forcing one search method to do everything.

Common Hybrid Search Mistakes

Combining Raw Scores Blindly

BM25 and embedding scores may have very different scales.

Use normalization or rank-based fusion such as RRF.

Using Different Embedding Models

Documents and queries must be represented consistently.

For dense-vector kNN search, Elasticsearch notes that query vectors should use the same model and dimensions as the indexed document vectors.

Ignoring Exact Matches

Semantic retrieval should not push an exact product ID or error code below loosely related content.

That is precisely why the BM25 branch matters.

Retrieving Too Many Candidates

A huge candidate pool can increase latency without significantly improving relevance.

Skipping Evaluation

Hybrid search is not automatically better for every dataset.

Test it against real queries.

Final Hybrid Search Flow

A practical implementation looks like this:

User Query
    ↓
┌─────────────────────────┐
│                         │
↓                         ↓
BM25                    Embedding
Search                   Model
↓                         ↓
Lexical Results       Vector Search
│                         │
└────────────┬────────────┘
             ↓
             RRF
             ↓
      Metadata Filters
             ↓
      Final Ranked Results

Final Thoughts

Keyword and semantic search solve different problems.

BM25 is strong when words themselves matter. Embeddings are strong when meaning matters.

Combining them gives you both:

BM25
Exact lexical relevance
        +
Embeddings
Semantic understanding
        ↓
Hybrid Search

A practical hybrid search BM25 embeddings architecture therefore starts with two independent retrieval paths and combines their rankings using a method such as RRF.

Start simple, test against real queries, and tune only after you understand where your current ranking fails.

That usually produces a better search system than relying entirely on either keyword search or embeddings alone.

Implementing Skip Links and Focus Management Properly

Samcorp — Tue, 01 Sep 2026 10:47:00 GMT

Keyboard accessibility is not just about making buttons respond to the Tab key.

Users also need an efficient way to bypass repeated navigation, understand where keyboard focus is, and maintain their position when dialogs, forms, or dynamic content change.

Good focus management accessibility makes keyboard navigation predictable instead of forcing users to repeatedly search for their place.

In this tutorial, we'll implement skip links, visible focus states, modal focus handling, form-error focus, and focus behavior for dynamic pages.

Why Focus Management Matters

Imagine navigating a site without a mouse.

Every page contains:

Logo
↓
Navigation Link 1
↓
Navigation Link 2
↓
Navigation Link 3
↓
More Header Controls
↓
Finally, Main Content

Repeating this on every page becomes frustrating.

WCAG 2.2 Success Criterion 2.4.1 Bypass Blocks requires a mechanism for bypassing repeated content, while 2.4.3 Focus Order requires keyboard focus to move in an order that preserves meaning and usability.

Step 1: Add a Skip Link

Place the skip link near the beginning of the page:


  
    Skip to main content
  

  
    
  

  
    Dashboard

When activated, the link moves the user past repeated navigation and directly to the primary content.

This is one of the standard techniques recommended by W3C for bypassing repeated page sections.

Step 2: Hide the Skip Link Until It Receives Focus

The skip link does not need to stay visually prominent all the time.

Hide it off-screen and reveal it when keyboard focus reaches it:

.skip-link {
  position: absolute;
  left: 1rem;
  top: -100px;
  padding: 0.75rem 1rem;
  background: #ffffff;
  color: #111111;
  z-index: 1000;
}

.skip-link:focus {
  top: 1rem;
}

Now a keyboard user can press Tab and immediately see:

[ Skip to main content ]

Avoid hiding the link with:

display: none;

or:

visibility: hidden;

because that removes it from keyboard navigation.

Step 3: Make the Target Reliably Focusable

For applications where you explicitly manage focus, you can make the main region programmatically focusable:


  Dashboard

tabindex="-1" allows JavaScript or fragment navigation to focus the element without placing it in the normal Tab sequence.

This distinction is important:

tabindex="0"
→ Included in normal Tab navigation

tabindex="-1"
→ Programmatically focusable

tabindex="1" or higher
→ Avoid

WAI's keyboard guidance strongly advises against using positive tabindex values to control page focus order.

Instead, keep the DOM structure itself logical.

Step 4: Keep Focus Indicators Visible

Never remove focus styling without replacing it.

This is problematic:

button:focus {
  outline: none;
}

A keyboard user may no longer know which button is active.

Use a clear focus indicator instead:

button:focus-visible,
a:focus-visible,
input:focus-visible {
  outline: 3px solid currentColor;
  outline-offset: 3px;
}

:focus-visible is useful because browsers can show the stronger indicator when keyboard-style focus needs to be communicated.

Accessible focus states should also be considered during interface design, not only during development. Well-planned UI/UX design can help teams build keyboard-friendly interfaces, accessible components, and WCAG-aligned user experiences.

Step 5: Manage Focus When Opening a Dialog

Dynamic interfaces require more deliberate focus management.

Suppose a user activates:

[Delete Account]

and a confirmation dialog opens.

Focus should not remain somewhere behind the dialog.

Using the native

element:




  Delete account?

JavaScript:

const dialog =
  document.querySelector("#confirm-dialog");

const openButton =
  document.querySelector("#open-dialog");

const closeButton =
  document.querySelector("#close-dialog");

openButton.addEventListener("click", () => {
  dialog.showModal();
});

closeButton.addEventListener("click", () => {
  dialog.close();
});

dialog.addEventListener("close", () => {
  openButton.focus();
});

The important flow is:

Delete Button
     ↓
Dialog Opens
     ↓
Focus Moves Into Dialog
     ↓
User Completes/Closes Dialog
     ↓
Focus Returns to Trigger

WAI guidance emphasizes maintaining predictable focus and restoring it logically when focused content disappears or closes.

For complex custom dialogs, use a proven accessible component pattern rather than inventing your own focus trap.

Step 6: Handle Focus After Content Is Removed

Suppose a task list contains:

Task A
Task B [Delete]
Task C

If the user deletes Task B, its delete button disappears.

Leaving focus on a removed DOM element can cause the browser to fall back to the document body, making keyboard navigation confusing.

Move focus somewhere logical instead:

deleteButton.addEventListener("click", () => {
  const nextTask =
    task.nextElementSibling ||
    task.previousElementSibling;

  task.remove();

  const nextButton =
    nextTask?.querySelector("button");

  nextButton?.focus();
});

The exact destination depends on the interface, but the principle remains:

After an action changes the UI, focus should land somewhere that makes sense.

WAI specifically identifies removed focused elements as situations where developers need to manage focus.

Step 7: Manage Focus After Form Errors

Consider a long registration form.

If validation fails, showing red borders alone may not help keyboard or screen-reader users understand what happened.

Create an error summary:


  There are 2 errors

  
    
      
        Enter a valid email address

Then move focus to it after validation:

const errorSummary =
  document.querySelector("#error-summary");

errorSummary.focus();

The user immediately learns that validation failed and can follow links to individual fields.

Step 8: Manage Focus in Single-Page Applications

Traditional navigation usually gives users a clear page transition.

Single-page applications can replace content without producing the same browser behavior.

For example:

Products
   ↓
User clicks "Account"
   ↓
URL changes
   ↓
New content appears

If focus remains on the old navigation link, a screen-reader or keyboard user may not immediately know that the page changed.

One approach is to focus the new page heading:


  My Account

After navigation:

document
  .querySelector("#page-heading")
  .focus();

A sensible SPA pattern is therefore:

Route Changes
      ↓
Update Page Content
      ↓
Update Document Title
      ↓
Move Focus to Main Heading

Do not move focus after every small UI update.

Use programmatic focus when it helps users understand a meaningful context change.

Step 9: Keep Focused Elements Visible

Sticky headers, cookie banners, and fixed toolbars can accidentally cover the element that receives focus.

For example:

┌──────────────────────────┐
│     Sticky Header        │
├──────────────────────────┤
│ Hidden Focused Button    │
│                          │
│ Main Content             │
└──────────────────────────┘

CSS such as this can help with anchored content:

html {
  scroll-padding-top: 6rem;
}

WCAG 2.2 added 2.4.11 Focus Not Obscured (Minimum) at Level AA, requiring a focused component not to be entirely hidden by author-created content.

Test sticky UI carefully with keyboard navigation.

Step 10: Use Native HTML Before ARIA

Do not make everything manually focusable.

Instead of:


  Save

prefer:

Native controls already provide much of the expected:

Keyboard behavior
Focus handling
Semantics
Browser support
Assistive-technology integration

Custom ARIA widgets require developers to implement their expected keyboard behavior themselves.

WAI's Authoring Practices explicitly notes this responsibility.

Consistent keyboard-friendly web development helps teams implement semantic HTML, logical focus order, screen-reader support, and accessible interactive components across the application.

Use ARIA when necessary, not as a replacement for semantic HTML.

A Practical Focus Management Checklist

Before shipping an interface, test it using only your keyboard:

Can I reach every interactive control?
        ↓
Is the focus indicator always visible?
        ↓
Does Tab order make sense?
        ↓
Can I bypass repeated navigation?
        ↓
Does focus enter dialogs correctly?
        ↓
Does focus return when dialogs close?
        ↓
Does dynamic content preserve my position?
        ↓
Are focused elements visible?

If any answer is no, the interface probably needs more focus-management work.

Common Accessibility Mistakes

Avoid these patterns:

Removing outline without providing another focus style
Using positive tabindex values to reorder controls
Moving focus unnecessarily
Forgetting to restore focus after closing dialogs
Allowing focus to remain on deleted elements
Hiding skip links with display: none
Creating clickable
elements instead of buttons
Allowing sticky content to cover focused controls
Testing accessibility only with a mouse

WAI recommends that all interactive functionality remain keyboard operable and that focus movement stay visible and predictable.

Final Thoughts

Good focus management accessibility is mostly about predictability.

Keyboard users should always be able to answer three questions:

Where am I? What can I do here? Where will I go next?

Start with semantic HTML and a correctly implemented skip link.

Preserve the browser's natural focus order wherever possible, provide strong visible focus states, and move focus programmatically only when an interface change genuinely requires it.

Skip links may look like a small accessibility feature, but combined with thoughtful focus management, they make complex websites significantly easier to navigate.

Building a Headless Shopify Storefront With Hydrogen

Samcorp — Tue, 01 Sep 2026 06:09:56 GMT

Shopify's traditional storefront works well for many stores, while businesses with more advanced requirements may need specialized Shopify development and greater control over performance, frontend architecture, design, or integrations.

That is where headless commerce comes in. Shopify manages products, inventory, customers, carts, and checkout, while your frontend runs separately.

For Shopify, Hydrogen is the official React-based framework for building headless storefronts. Current Hydrogen projects are built on React Router and include Shopify-optimized components, utilities, routing, Storefront API access, and deployment support.

In this Shopify Hydrogen tutorial, we will create a storefront, connect it to Shopify, fetch product data, understand cart handling, and deploy it with Oxygen.

What Are Hydrogen and Oxygen?

The basic architecture looks like this:

Customer
   ↓
Hydrogen Storefront
   ↓
Shopify Storefront API
   ↓
Products / Collections / Cart / Checkout

Hydrogen handles your storefront application.

Shopify Storefront API exposes commerce data through GraphQL. Shopify's current Storefront API is GraphQL-only and supports products, collections, carts, checkout-related commerce flows, and more.

Oxygen is Shopify's hosting environment designed for Hydrogen storefronts.

You can technically host Hydrogen elsewhere, but Oxygen gives you a straightforward Shopify-native deployment path.

Step 1: Create a Hydrogen Project

Make sure you have a recent Node.js environment installed.

Create a starter project with:

npm create @shopify/hydrogen@latest -- --quickstart

Shopify's current quickstart creates a working storefront using Mock.shop, so you can experiment before connecting a real store.

It also generates common routes for:

Products
Collections
Cart
Accounts
Search
Policies
Blogs
Other storefront pages

Enter the project:

cd hydrogen-quickstart

Start development:

shopify hydrogen dev

Then open:

http://localhost:3000

You should see the starter storefront running with sample products.

Step 2: Understand the Project Structure

A Hydrogen application contains familiar React application files:

app/
├── components/
├── routes/
├── styles/
├── root.jsx
├── entry.client.jsx
└── entry.server.jsx

The routes directory is especially important.

Common storefront URLs include:

/
/products/:handle
/collections
/collections/:handle
/cart
/account
/search

Hydrogen uses route loaders to fetch Shopify data on the server before rendering the page.

Shopify recommends this pattern for Storefront API, Customer Account API, and third-party data fetching.

Step 3: Connect Hydrogen to Your Shopify Store

The starter initially uses Mock.shop data.

To connect a real Shopify store, run:

npx shopify hydrogen link

Follow the prompts to log into Shopify and choose or create a Hydrogen storefront.

Next, pull the required environment configuration:

npx shopify hydrogen env pull

Shopify automatically adds values such as your Storefront API credentials and Customer Account API settings to the local environment.

Restart the development server:

npm run dev

Your storefront should now display products from your Shopify store.

Step 4: Fetch Products With the Storefront API

One of Hydrogen's biggest advantages is its built-in Storefront API client.

A basic product route can use a loader like this:

import {useLoaderData} from '@shopify/remix-oxygen';

export async function loader({params, context}) {
  const {handle} = params;

  const {product} = await context.storefront.query(
    PRODUCT_QUERY,
    {
      variables: {handle},
    },
  );

  if (!product) {
    throw new Response('Product not found', {
      status: 404,
    });
  }

  return {product};
}

export default function ProductPage() {
  const {product} = useLoaderData();

  return (
    
      {product.title}
      {product.description}
    
  );
}

Now define the GraphQL query:

const PRODUCT_QUERY = `#graphql
  query Product($handle: String!) {
    product(handle: $handle) {
      id
      title
      description
      featuredImage {
        url
        altText
        width
        height
      }
    }
  }
`;

The flow is:

/products/example-product
          ↓
Route Loader
          ↓
Storefront API
          ↓
GraphQL Product Query
          ↓
React Component

Hydrogen's built-in storefront.query() client is designed specifically for this server-side Storefront API pattern.

Step 5: Render Product Information

You can now expand the page with an image and product information:

export default function ProductPage() {
  const {product} = useLoaderData();

  return (
    
      {product.title}

      {product.featuredImage && (
        
      )}

      {product.description}
    
  );
}

In a production storefront, you would normally add:

Product variants
Pricing
Availability
Quantity controls
Add-to-cart button
Product recommendations
Structured SEO metadata

The important part is that Shopify remains the commerce backend while Hydrogen controls the customer-facing experience.

Step 6: Handle the Shopping Cart

The Hydrogen starter already includes cart routes and Shopify cart utilities.

A simplified add-to-cart interaction works around a Shopify merchandise variant ID:

Behind the scenes, Hydrogen sends the appropriate cart mutation to Shopify.

Your architecture remains:

Product Page
     ↓
Add to Cart
     ↓
Hydrogen Cart Handler
     ↓
Storefront API
     ↓
Shopify Cart

Shopify's Hydrogen cart utilities support adding, updating, and removing line items while keeping commerce logic connected to the Storefront API.

Step 7: Think About Performance

Going headless does not automatically make a storefront faster.

You still need to manage:

GraphQL query size
Image optimization
Caching
JavaScript bundle size
Third-party scripts
Server response times

Hydrogen supports caching Shopify API responses, and Shopify's current tooling provides caching options for non-personalized catalog data such as products, collections, and pages.

For example:

const {product} = await storefront.query(
  PRODUCT_QUERY,
  {
    variables: {handle},
    cache: storefront.CacheLong(),
  },
);

Use longer caching for stable public catalog data and avoid public caching for personalized customer information.

Step 8: Keep SEO in the Architecture

A headless storefront still needs strong ecommerce SEO, especially because your team becomes responsible for URL structure, metadata, canonicalization, structured data, redirects, and crawlability.

Preserve predictable URLs such as:

/products/:handle
/collections/:handle

Also implement:

Unique page titles
Meta descriptions
Canonical URLs
Product structured data
XML sitemap
Robots.txt
Redirects from old URLs

This becomes especially important when migrating an existing Shopify theme to Hydrogen.

Shopify recommends maintaining redirects when storefront URL structures change so existing links continue to work.

Step 9: Deploy the Hydrogen Storefront

Once the storefront works locally, deploy it to Oxygen.

First, make sure the project is linked to your Shopify storefront.

Then run:

npx shopify hydrogen deploy

Choose the preview environment when prompted.

The Hydrogen CLI builds the application, creates an Oxygen deployment, and returns a preview URL.

Your final architecture becomes:

Customer
   ↓
Custom Domain
   ↓
Hydrogen on Oxygen
   ↓
Shopify Storefront API
   ↓
Shopify Commerce Backend

When Does Hydrogen Make Sense?

Hydrogen is worth considering when you need more frontend control than a standard Shopify theme provides.

Typical use cases include:

Highly customized ecommerce experiences
Complex frontend integrations
Content-rich commerce
Multiple backend data sources
Custom product discovery
International storefront experiences
Performance-sensitive ecommerce applications

However, headless also introduces more frontend ownership.

You become responsible for areas such as:

Routing
SEO implementation
Testing
Monitoring
Accessibility
Frontend architecture
Deployment workflows

Do not choose headless simply because it sounds more modern.

Choose it when the additional flexibility solves a real business or technical requirement.

Common Hydrogen Mistakes

Avoid these problems when following a Shopify Hydrogen tutorial or building a production storefront:

Fetching more GraphQL fields than the page requires
Ignoring caching
Building custom cart logic when Hydrogen utilities already solve it
Exposing private API credentials to browser code
Forgetting redirects during migration
Ignoring SEO because the storefront is headless
Recreating Shopify functionality unnecessarily
Adding too many client-side dependencies

Keep the architecture simple until the store genuinely requires additional complexity.

Final Thoughts

Building a headless Shopify storefront with Hydrogen gives developers much more control over the frontend while keeping Shopify responsible for the commerce engine.

The practical workflow is straightforward:

Create Hydrogen Project
        ↓
Connect Shopify Store
        ↓
Query Storefront API
        ↓
Build Product Experience
        ↓
Configure Cart
        ↓
Optimize Performance + SEO
        ↓
Deploy to Oxygen

Hydrogen is most valuable when you need a storefront that goes beyond the limits of a conventional Shopify theme.

Start with Shopify's generated storefront, understand its existing routes and commerce utilities, and customize only the areas that genuinely need a headless architecture.

Practical Feature Flags for Safe Continuous Deployment

Samcorp — Mon, 31 Aug 2026 09:45:41 GMT

Continuous deployment helps teams release software faster, but pushing every new feature to all users at once can be risky.

Feature flags reduce that risk by separating deployment from release. You can deploy code to production, keep the feature turned off, enable it for a small audience, monitor real-world behavior, and expand the rollout gradually.

That makes feature flags deployment a practical way to reduce release risk without slowing down software delivery.

What Is a Feature Flag?

A feature flag controls whether a feature is enabled or disabled at runtime.

Instead of calling a new feature directly:

showNewCheckout();

you can place it behind a feature flag:

if (newCheckoutEnabled) {
  showNewCheckout();
} else {
  showCurrentCheckout();
}

The release process then becomes:

Deploy Code
    ↓
Feature OFF
    ↓
Enable for Small Group
    ↓
Monitor
    ↓
Increase Rollout
    ↓
Release to Everyone

This gives teams more control over how and when users receive new functionality.

Step 1: Add a Feature Flag

For this example, we can use OpenFeature, an open standard for feature flagging.

Install the SDK:

npm install @openfeature/server-sdk

Then evaluate the feature flag in your application:

const enabled =
  await featureFlags.getBooleanValue(
    "new-checkout",
    false
  );

if (enabled) {
  return renderNewCheckout();
}

return renderCurrentCheckout();

The false value acts as a safe fallback if the feature flag cannot be evaluated.

This means the application continues using the existing checkout experience instead of exposing an unfinished or potentially risky feature.

Step 2: Deploy Before Releasing

One of the biggest advantages of feature flags is the ability to deploy code without immediately releasing the feature.

Deploy the new application version while keeping the flag disabled:

New Version
    ↓
Deploy to Production
    ↓
Feature Flag = OFF
    ↓
Existing Experience Continues

The new code is now running in production, but users still see the existing experience.

This approach is especially useful when releasing modern cloud applications, where teams may want to validate production stability before exposing a new capability to all users.

Step 3: Enable the Feature for Selected Users

Instead of enabling a feature for everyone at once, start with a controlled group.

For example:

Internal team members
Beta users
Selected customer accounts
Specific regions
Particular subscription plans

You can also provide evaluation context to the feature flag system:

const context = {
  targetingKey: user.id,
  region: user.region,
  plan: user.plan,
};

The flag provider can then use this information to determine which users should receive the new feature.

For example, you could release a feature only to internal employees or premium customers before making it generally available.

Step 4: Use Progressive Rollouts

Once internal testing is successful, increase exposure gradually.

A rollout could look like this:

Internal Users
      ↓
      5%
      ↓
     10%
      ↓
     25%
      ↓
     50%
      ↓
    100%

A practical rollout plan might be:

Stage	Traffic
Internal Testing	Employees
Canary	5%
Early Rollout	10%
Wider Rollout	25–50%
General Release	100%

This approach makes it easier to detect problems before they affect every user.

If an issue appears when only 5% of traffic is using the feature, the impact is much smaller than discovering the same issue after a full release.

Step 5: Monitor the Rollout

Do not increase the rollout percentage without checking production metrics.

Important metrics may include:

Error rate
Response time
CPU usage
Memory usage
Database load
Failed transactions
Conversion rate
User engagement

For example:

Feature Rollout: 10%

Error Rate
Before: 0.3%
After:  0.4%

Latency
Before: 220 ms
After:  235 ms

If the system remains healthy, continue increasing the rollout.

If error rates, latency, failed transactions, or other important metrics increase significantly, disable the feature flag and investigate the problem.

This is one of the main benefits of progressive delivery: teams can react without performing another deployment.

Step 6: Use Flags as Kill Switches

Feature flags can also work as operational kill switches for risky integrations or services.

For example:

const paymentsEnabled =
  await featureFlags.getBooleanValue(
    "payments-enabled",
    true
  );

if (!paymentsEnabled) {
  return showMaintenanceMessage();
}

return processPayment();

If the payment service becomes unstable, the feature can be disabled without rebuilding or redeploying the application.

Kill switches can be useful for:

External APIs
Background jobs
Payment systems
Expensive features
Experimental services
Third-party integrations

This can reduce recovery time during production incidents.

Step 7: Connect Flags to Monitoring

A good feature flag system should help teams answer questions such as:

Which variation did the user receive?

When was the flag changed?

Did errors increase after rollout?

What percentage of users currently has the feature enabled?

Feature flags become far more valuable when connected to logging, monitoring, and observability tools.

For example, application logs can record which flag variation a user received when an error occurred.

This helps teams determine whether a production issue started after a rollout change.

A useful event might contain information such as:

User: 48291
Feature: new-checkout
Variation: enabled
Rollout: 25%
Timestamp: 14:32 UTC

When rollout data and application telemetry are connected, troubleshooting becomes much faster.

Step 8: Remove Old Flags

Temporary feature flags should not remain in the application forever.

After a feature reaches 100% rollout and remains stable, remove the flag and the unused code.

A simple cleanup process looks like this:

100% Rollout
     ↓
Monitor Stability
     ↓
Remove Old Code
     ↓
Remove Flag

For example, avoid keeping logic like this permanently:

if (newCheckout) {
  newCheckoutFlow();
} else {
  oldCheckoutFlow();
}

Once the old checkout flow is no longer required, remove both the feature flag and the unused code.

Otherwise, large numbers of old flags can make the application difficult to understand and maintain.

Feature flag cleanup should therefore be part of the release process.

Recommended Feature Flag Deployment Flow

A practical feature flag deployment process looks like this:

Build Feature
     ↓
Add Feature Flag
     ↓
Deploy with Flag OFF
     ↓
Enable Internally
     ↓
Roll Out to 5%
     ↓
Monitor
     ↓
25% → 50% → 100%
     ↓
Monitor Stability
     ↓
Remove Temporary Flag

This approach gives teams several opportunities to stop or reverse a release before it affects the entire user base.

Common Mistakes to Avoid

Feature flags are useful, but poor implementation can create new problems.

Avoid these common mistakes:

Enabling a feature for everyone immediately
Using unsafe fallback values
Rolling out without monitoring
Giving too many people permission to change production flags
Keeping temporary flags forever
Creating deeply nested flag logic
Using flags without documenting ownership
Changing rollout percentages without tracking the change
Treating feature flags as a replacement for testing

Feature flags reduce deployment risk, but they should work alongside automated testing, CI/CD controls, monitoring, and observability.

They are an additional release-control mechanism, not a substitute for good engineering practices.

Feature Flags vs Traditional Deployment

Traditional releases often connect deployment and release together:

Deploy New Version
      ↓
Everyone Gets Feature

Feature flags separate those activities:

Deploy New Version
      ↓
Feature Disabled
      ↓
Controlled Release
      ↓
Monitor
      ↓
Increase Exposure

That separation makes production releases easier to control and reverse.

If something goes wrong, teams may be able to disable the feature immediately rather than rolling back the entire application.

A disciplined software development process should combine feature flags with automated testing, CI/CD controls, monitoring, and clear release ownership.

Final Thoughts

A strong feature flags deployment strategy separates shipping code from releasing features.

Deploy the code first, keep the new feature disabled, enable it for a small audience, monitor real production behavior, and increase exposure gradually.

The most effective feature flag systems include safe defaults, targeted releases, progressive rollouts, kill switches, monitoring, access controls, and regular cleanup.

Feature flags do not remove deployment risk completely, but they give engineering teams much better control over how that risk reaches users.

Building a CI/CD Pipeline That Enforces Security Gates

Samcorp — Mon, 31 Aug 2026 09:16:15 GMT

A CI/CD pipeline should not deploy code just because the build succeeds. It should also verify that the application meets basic security requirements before reaching production.

That is where CI/CD security gates help.

A security gate is an automated checkpoint that blocks the pipeline when it detects problems such as vulnerable dependencies, insecure code, failed tests, or unsafe container images.

In this tutorial, we will create a simple GitHub Actions pipeline that adds security checks before deployment.

What Are CI/CD Security Gates?

A normal CI/CD pipeline may look like this:

Code Push
   ↓
Build
   ↓
Test
   ↓
Deploy

A secure pipeline adds additional checkpoints:

Code Push
   ↓
Build & Test
   ↓
Dependency Scan
   ↓
SAST
   ↓
Container Scan
   ↓
Production Approval
   ↓
Deploy

If an important security check fails, the deployment should stop automatically.

The goal is simple:

Unsafe code should never reach production by accident.

Step 1: Create the Base CI Pipeline

Create the following GitHub Actions workflow file:

.github/workflows/secure-pipeline.yml

Start with a basic build and testing job:

name: Secure CI/CD Pipeline

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Code
        uses: actions/checkout@v7

      - name: Setup Node.js
        uses: actions/setup-node@v7
        with:
          node-version: 24
          cache: npm

      - name: Install Dependencies
        run: npm ci

      - name: Run Tests
        run: npm test

If the tests fail, this job fails and the pipeline should not continue toward deployment.

This gives us our first security-related gate: code that cannot pass its tests cannot move forward.

Step 2: Add a Dependency Security Gate

Third-party packages can introduce known vulnerabilities into your application.

For a Node.js project, you can add a dependency audit:

dependency-audit:
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v7

    - uses: actions/setup-node@v7
      with:
        node-version: 24

    - run: npm ci

    - name: Check Vulnerable Dependencies
      run: npm audit --audit-level=high

With this configuration, high-severity dependency vulnerabilities can cause the job to fail.

For pull requests, you can also use GitHub's dependency review:

dependency-review:
  if: github.event_name == 'pull_request'
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v7

    - name: Dependency Review
      uses: actions/dependency-review-action@v5
      with:
        fail-on-severity: high

This helps prevent developers from introducing risky dependencies without noticing them.

Instead of discovering the problem after deployment, the pipeline catches it during development.

Step 3: Add Static Security Testing

Static Application Security Testing, commonly called SAST, analyzes source code for potential security weaknesses.

GitHub CodeQL can be added directly to the workflow:

sast:
  runs-on: ubuntu-latest

  permissions:
    contents: read
    security-events: write

  steps:
    - uses: actions/checkout@v7

    - name: Initialize CodeQL
      uses: github/codeql-action/init@v4
      with:
        languages: javascript-typescript

    - name: Analyze Code
      uses: github/codeql-action/analyze@v4

This gives the pipeline another security checkpoint before deployment.

For stronger enforcement, configure repository rules so required security checks must pass before a pull request can merge.

That way, security scanning becomes part of the development workflow rather than an optional report developers can ignore.

Step 4: Scan the Container Image

If your application runs inside Docker, you should also scan the final container image.

A container may contain vulnerabilities even when the application source code itself looks secure.

Tools such as Trivy can inspect operating-system packages and application dependencies inside an image.

For example:

trivy image \
  --exit-code 1 \
  --severity HIGH,CRITICAL \
  my-app:${GITHUB_SHA}

The important option is:

--exit-code 1

It tells the scanner to return a failed status when serious vulnerabilities are detected.

Without this setting, your pipeline may successfully detect vulnerabilities but continue deploying anyway.

That is not really a security gate.

It is only a security report.

A real security gate must be able to stop the pipeline.

Step 5: Connect Security Gates to Deployment

Now make the deployment job depend on the security jobs:

deploy:
  needs:
    - test
    - dependency-audit
    - sast
    - container-scan

  runs-on: ubuntu-latest

  environment:
    name: production

  steps:
    - name: Deploy Application
      run: ./scripts/deploy.sh

The pipeline now behaves like this:

Tests ─────────────┐
Dependencies ──────┤
SAST ──────────────┼──> Production Deployment
Container Scan ────┘

If one required job fails, the deployment job will not start.

For example:

Test Passed
Dependency Scan Passed
SAST Passed
Container Scan Failed
        ↓
Deployment Blocked

This is the core idea behind effective CI/CD security gates.

Deployment should happen only when every required security condition has been satisfied.

Step 6: Add Production Approval

Automated checks are valuable, but sensitive production deployments may also require human approval.

GitHub environments can be configured with required reviewers.

The release process can then look like this:

Security Checks
      ↓
All Passed
      ↓
Manual Approval
      ↓
Production

This creates another safeguard for important releases.

Automated tools determine whether the software meets technical requirements, while an authorized person can verify whether the release should actually proceed.

This can be especially useful for:

High-risk production changes
Financial applications
Healthcare systems
Enterprise platforms
Infrastructure changes
Major releases

Human approval should not replace automated security checks. It should complement them.

Step 7: Protect Deployment Credentials

Security checks are not enough if the deployment credentials themselves are poorly protected.

Avoid storing long-lived cloud credentials inside GitHub secrets whenever possible.

For example, try to avoid permanently storing credentials such as:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY

When building or deploying modern cloud applications, use OpenID Connect (OIDC) whenever your cloud platform supports it instead of relying on long-lived credentials.A workflow can request temporary credentials using permissions such as:

permissions:
  contents: read
  id-token: write

The cloud provider can then issue short-lived credentials specifically for that workflow execution.

Short-lived credentials reduce the potential damage if credentials are exposed or compromised.

Protect the Pipeline Too

Security gates are only useful when attackers cannot easily modify or bypass them.

Your CI/CD configuration should therefore be treated as security-sensitive code.

Protect workflow files with pull-request reviews and branch protection rules.

You should also keep GitHub Actions permissions as limited as possible.

For example:

permissions:
  contents: read

Then grant additional permissions only to the jobs that genuinely need them.

For security-sensitive projects, also consider pinning third-party GitHub Actions to reviewed commit SHAs rather than relying only on movable version tags.

The pipeline itself is part of your application's attack surface.

Recommended CI/CD Security Flow

A practical secure delivery pipeline can look like this:

Developer Push
      ↓
Build & Tests
      ↓
Dependency Scan
      ↓
SAST
      ↓
Container Scan
      ↓
All Gates Passed?
     /        \
   No          Yes
   ↓            ↓
 Block      Approval
                ↓
            Production

Each stage answers a different security question:

Security Gate	What It Checks
Build and Tests	Does the application work correctly?
Dependency Scan	Are known vulnerable packages present?
SAST	Does the source code contain security weaknesses?
Container Scan	Does the deployment image contain serious vulnerabilities?
Production Approval	Is the release authorized for production?

Do not add security tools simply because they are popular.

Every gate should protect against a real risk and have a clear rule defining when the pipeline must stop.

What Makes a Good Security Gate?

A useful security gate should have three characteristics.

1. It Checks a Meaningful Risk

The gate should detect something that could realistically affect your application.

For example:

Critical dependency vulnerabilities
Authentication flaws
Secrets committed to source control
Dangerous container vulnerabilities
Failed security tests

2. It Has a Clear Failure Threshold

Teams should know exactly what causes the gate to fail.

For example:

Critical vulnerability → Block deployment
High vulnerability → Block deployment
Medium vulnerability → Review or warning
Low vulnerability → Track for remediation

The exact policy will depend on your application's risk level.

3. It Actually Stops the Pipeline

A scanner that produces a report but always exits successfully is not enforcing anything.

Security tools should return a failing exit code when your organization's defined security threshold is exceeded.

That turns security scanning into a real deployment control.

Final Thoughts

Strong CI/CD security gates make security part of software delivery instead of leaving it until the end of the release cycle.

Start with a few meaningful checks:

Automated testing
Dependency scanning
Static security testing
Container image scanning
Production approval

As your application and security requirements grow, these practices can become part of a broader digital transformation strategy focused on secure, automated, and scalable software delivery.

The goal is not to create the most complicated CI/CD pipeline.

The goal is to make sure serious security problems stop a release before they reach production.

Migrating REST to GraphQL Incrementally

Samcorp — Mon, 31 Aug 2026 07:21:58 GMT

Migrating from REST to GraphQL does not mean you need to rewrite your entire API at once.

A safer REST to GraphQL migration is to introduce GraphQL gradually, keep existing REST services running, and move one feature at a time.

This approach reduces production risk and gives teams enough time to test performance, authentication, caching, and client behavior.

Why Migrate Incrementally?

A full REST replacement can affect:

Backend services
Frontend applications
Authentication
Caching
Monitoring
Third-party integrations

Instead of changing everything together, migrate in small steps.

For example:

User Profile → Orders → Products → Mutations → Remaining APIs

Each phase can be tested and rolled back independently.

Step 1: Review Existing REST APIs

Start by identifying the endpoints your application actually uses.

Example:

GET /users/:id
GET /users/:id/orders
POST /orders

Choose a simple and low-risk endpoint first.

A read-only feature such as a user profile is usually better than starting with payments or other critical transactions.

Step 2: Design the GraphQL Schema

Avoid copying REST endpoints directly into GraphQL.

Instead of:

type Query {
  getUser(id: ID!): User
  getUserOrders(id: ID!): [Order]
}

Use relationships:

type User {
  id: ID!
  name: String!
  email: String!
  orders: [Order!]!
}

type Order {
  id: ID!
  total: Float!
  status: String!
}

type Query {
  user(id: ID!): User
}

Now the frontend works with business objects instead of individual endpoints.

Step 3: Add GraphQL Over Existing REST

You do not need to remove REST immediately.

GraphQL can act as a layer in front of your current APIs.

Frontend
   |
GraphQL API
   |
Existing REST APIs

For example, a resolver can call the current REST endpoint:

const resolvers = {
  Query: {
    user: async (_, { id }) => {
      const response = await fetch(
        `https://api.example.com/users/${id}`
      );

      return response.json();
    },
  },
};

Your existing backend continues working while clients slowly move to GraphQL.

For larger modernization projects, professional custom API development and integration can help teams design, secure, and gradually migrate REST APIs to GraphQL.

Step 4: Migrate One Frontend Feature

Suppose your application currently makes two REST calls:

GET /users/42
GET /users/42/orders

With GraphQL, the client can request both together:

query GetUser {
  user(id: "42") {
    id
    name
    email
    orders {
      id
      total
      status
    }
  }
}

Start by migrating only one page or workflow.

Other parts of the application can continue using REST.

For applications that require scalable frontend and backend integration, modern web development can combine GraphQL, REST APIs, and application architecture within the same development stack.

Step 5: Use Feature Flags

Feature flags make migrations easier to control.

if (features.useGraphQL) {
  return loadWithGraphQL();
}

return loadWithREST();

You can gradually enable GraphQL for:

Internal users
↓
Beta users
↓
25% traffic
↓
50% traffic
↓
100% traffic

If something goes wrong, switch back to REST quickly.

Step 6: Compare REST and GraphQL Results

Before fully switching traffic, compare both implementations.

Check for:

Missing fields
Incorrect values
Permission differences
Null handling
Pagination issues
Error differences

This is especially useful for read operations.

Avoid sending the same write through both REST and GraphQL because it may create duplicate records.

Step 7: Keep Authentication Consistent

If REST currently uses:

Authorization: Bearer

the GraphQL layer should forward the same identity to downstream services.

During early migration, keeping existing authorization rules reduces unnecessary risk.

You can redesign authorization later once the GraphQL layer is stable.

Step 8: Watch for the N+1 Problem

GraphQL can accidentally create too many backend requests.

For example:

query {
  users {
    name
    orders {
      id
    }
  }
}

If you have 100 users, a poorly designed resolver might make 100 separate requests for orders.

Use techniques such as:

Request batching
Caching
DataLoader
Batch REST endpoints

Always measure downstream traffic, not just frontend requests.

Step 9: Monitor the Migration

Track important metrics such as:

GraphQL error rate
REST traffic
API latency
Payload size
Backend request count
Cache performance
Client failures

Do not assume GraphQL is faster just because the browser makes fewer HTTP requests.

Measure the complete request path.

Step 10: Migrate Mutations Later

Once GraphQL reads are stable, start migrating write operations.

For example:

mutation CreateOrder($input: CreateOrderInput!) {
  createOrder(input: $input) {
    id
    status
  }
}

Initially, GraphQL can still call:

POST /orders

Later, the resolver can connect directly to the service layer.

The client does not need to know when the internal implementation changes.

Step 11: Deprecate REST Gradually

Do not remove REST endpoints immediately.

Monitor usage first.

Example:

Week 1 → 100,000 REST requests
Week 2 → 45,000
Week 3 → 8,000
Week 4 → 0

Confirm that all web apps, mobile apps, internal tools, and integrations have migrated.

Only then remove the endpoint.

Recommended Migration Flow

A practical REST to GraphQL migration looks like this:

Review REST APIs
      ↓
Choose one feature
      ↓
Design GraphQL schema
      ↓
Connect GraphQL to REST
      ↓
Migrate one client
      ↓
Test and monitor
      ↓
Optimize performance
      ↓
Migrate more features
      ↓
Deprecate unused REST APIs

Common Mistakes to Avoid

Avoid these common problems:

Rewriting the complete backend first
Copying REST endpoints directly into GraphQL
Migrating every frontend screen together
Ignoring N+1 requests
Changing authentication during the migration
Removing REST without checking traffic
Assuming GraphQL is automatically faster

Keep each migration step small and measurable.

Final Thoughts

A successful REST to GraphQL migration is not about replacing REST as quickly as possible.

Start with one useful GraphQL feature, keep existing REST services running, test the result, monitor production traffic, and gradually expand.

GraphQL can first work as a layer over REST and later connect directly to backend services.

This approach gives you the benefits of GraphQL without turning API modernization into a risky full-system rewrite.

Unreal Blueprints to C++: When and How to Convert

Samcorp — Fri, 28 Aug 2026 11:49:02 GMT

A practical guide to deciding which Unreal Engine Blueprint systems belong in C++, how to migrate them safely, and why converting everything is usually the wrong approach.

Blueprints are one of Unreal Engine's biggest productivity advantages.

You can prototype gameplay, connect events, expose designer controls, and test ideas without waiting on a traditional code-build cycle.

But as a project grows, a Blueprint that started like this:

Input
  ↓
Move Player
  ↓
Play Animation

can eventually become:

Event Tick
  ↓
Branch
  ↓
Loop
  ↓
Multiple Casts
  ↓
AI Logic
  ↓
Inventory Logic
  ↓
Movement Logic
  ↓
More Branches...

At that point, developers often ask:

Should I convert this Blueprint to C++?

The answer is usually not all of it.

A good Unreal Blueprint to C++ migration moves the right responsibilities into native code while keeping Blueprint where visual scripting still provides faster iteration.

Blueprint vs C++ Is Not an Either/Or Decision

Unreal Engine is designed for the two systems to work together.

A useful production architecture is:

        C++
         │
 ┌───────┴────────┐
 │ Core Systems   │
 │ Gameplay Logic │
 │ Algorithms     │
 │ Networking     │
 └───────┬────────┘
         │
         ▼
     Blueprints
         │
 ┌───────┴────────┐
 │ Configuration  │
 │ Events         │
 │ UI             │
 │ Designer Logic │
 └────────────────┘

This gives programmers control over architecture while designers retain Blueprint's rapid iteration.

That hybrid approach is also common in professional Unreal Engine game development, where C++ and Blueprints can serve different responsibilities rather than competing for ownership of the entire project.

When Should You Move Blueprint Logic to C++?

There are several good signals.

1. The Blueprint Has Become Too Complex

Large Blueprint graphs become difficult to navigate.

You may see:

Functions → Macros → Events → Casts
     ↓
More Functions
     ↓
Nested Logic
     ↓
More Dependencies

If developers spend more time understanding the graph than modifying it, moving the underlying system to C++ may improve maintainability.

C++ also works better with normal text-based source-control workflows, code reviews, diffs, and merges.

2. The Logic Runs Very Frequently

Blueprint code executes through Unreal's Blueprint virtual machine, while C++ is compiled into native machine code.

The difference matters most when you're performing substantial work repeatedly.

Examples include:

Large loops
Complex calculations
Many actors ticking
Large datasets
AI processing
Procedural systems
Simulation logic

Consider this Blueprint-style pattern:

Event Tick
   ↓
Get All Enemies
   ↓
For Each Enemy
   ↓
Calculate Distance
   ↓
Check Visibility
   ↓
Update State

With hundreds of actors, this can become expensive.

That logic may belong in C++—although eliminating unnecessary Tick work should often be your first optimization.

Profile Before You Convert

Don't rewrite Blueprints because someone says:

C++ is faster.

Measure first.

Use tools such as Unreal Insights to determine where frame time is actually going.

Your problem might be:

GPU rendering
Physics
Animation
Asset loading
Blueprint VM
Networking

If rendering is consuming most of the frame, converting an unrelated Blueprint won't magically improve performance.

A better workflow is:

Profile
   ↓
Identify Bottleneck
   ↓
Understand Cause
   ↓
Optimize
   ↓
Measure Again

Convert only when Blueprint execution or architecture is actually part of the problem.

What Should Usually Stay in Blueprint?

Blueprint remains excellent for:

designer-controlled gameplay
simple event sequences
UI behavior
animation integration
level scripting
configuration
prototyping
visual effects triggers

For example:

Player Enters Trigger
       ↓
Open Door
       ↓
Play Sound
       ↓
Trigger VFX

doesn't necessarily need a C++ rewrite.

Blueprint is doing exactly what it's good at: expressing straightforward gameplay behavior visually.

Understanding these strengths before migration is important; this overview of Blueprints in Unreal Engine development provides additional context around prototyping, visual scripting, and designer-friendly development.

A Better Migration Strategy

Suppose you currently have:

BP_PlayerCharacter

containing:

Movement
Health
Combat
Inventory
Interaction
Animation Events
VFX
Audio

Don't immediately recreate the entire Blueprint in C++.

Instead, create a native foundation.

APlayerCharacterBase (C++)
          │
          ▼
BP_PlayerCharacter

Move stable system-level logic into the C++ parent while allowing the Blueprint child to handle presentation and tuning.

Step 1: Add C++ to the Project

A Blueprint project doesn't need to be abandoned or recreated.

Add a new C++ class through Unreal Editor and choose the appropriate parent class.

For a character:

#include "GameFramework/Character.h"
#include "PlayerCharacterBase.generated.h"

UCLASS()
class MYGAME_API APlayerCharacterBase
    : public ACharacter
{
    GENERATED_BODY()

public:

    APlayerCharacterBase();

};

Adding a C++ class sets up the native code environment while your existing Blueprint assets can remain in the project.

Step 2: Move Stable Variables Into C++

Suppose Blueprint contains:

Health
MaxHealth
MovementSpeed
Damage

Move the structural variables into C++:

UPROPERTY(
    EditAnywhere,
    BlueprintReadWrite,
    Category="Player"
)
float MaxHealth = 100.0f;

UPROPERTY(
    BlueprintReadOnly,
    Category="Player"
)
float CurrentHealth;

The key is the Unreal reflection system.

UPROPERTY allows C++ to define the architecture while still exposing appropriate values to Blueprint.

Step 3: Move Core Functions

Suppose damage handling has grown into a complicated Blueprint graph.

Move the core calculation:

UFUNCTION(BlueprintCallable)
void ApplyPlayerDamage(float DamageAmount);

Implementation:

void APlayerCharacterBase::ApplyPlayerDamage(
    float DamageAmount)
{
    CurrentHealth -= DamageAmount;

    CurrentHealth = FMath::Clamp(
        CurrentHealth,
        0.0f,
        MaxHealth
    );
}

Blueprint can still call:

Apply Player Damage

as a normal node.

Now you get:

C++
 └── Damage calculation

Blueprint
 └── Animation
 └── Sound
 └── VFX
 └── UI feedback

This separation scales much better.

Step 4: Let Blueprint Implement Presentation

Sometimes C++ should define when something happens without deciding how it looks.

For example:

UFUNCTION(BlueprintImplementableEvent)
void OnPlayerDeath();

C++:

if (CurrentHealth <= 0.0f)
{
    OnPlayerDeath();
}

Blueprint can implement:

On Player Death
      ↓
Play Animation
      ↓
Spawn VFX
      ↓
Play Sound
      ↓
Show Game Over UI

This creates a useful boundary:

C++ owns the system. Blueprint owns presentation.

Step 5: Reparent the Existing Blueprint

Once the C++ base class exists, your Blueprint can inherit from it.

Conceptually:

Before

BP_PlayerCharacter
       ↓
   Character


After

BP_PlayerCharacter
       ↓
APlayerCharacterBase
       ↓
   Character

Then migrate Blueprint functionality gradually.

Don't move twenty systems at once.

Try:

Health
  ↓
Test

Movement
  ↓
Test

Combat
  ↓
Test

Inventory
  ↓
Test

Small migrations make regressions much easier to locate.

Blueprint Header View Can Help

Current Unreal Engine versions include Blueprint Header View.

You can right-click a Blueprint class or struct and use:

Preview Equivalent C++ Header

to see C++-style declarations representing Blueprint elements such as:

Variables
Functions
Components
Event Dispatchers

But there is an important limitation:

it does not automatically convert your entire Blueprint implementation into clean production C++.

Think of it as a migration aid.

You still need to implement and architect the actual C++ logic.

Don't Convert Everything

One of the easiest mistakes during an Unreal Blueprint to C++ migration is turning it into a rewrite.

You don't need:

Blueprint
    ↓
Delete Everything
    ↓
100% C++

A healthier target is often:

              GAME
               │
        ┌──────┴──────┐
        │             │
       C++        Blueprint
        │             │
Core systems       UI
Algorithms         VFX
Performance        Events
Networking         Tuning
Architecture       Content

The exact split depends on your project and team.

A Simple Decision Rule

Before converting a Blueprint, ask four questions.

Is it performance-critical?
        │
        YES → Consider C++

Is the graph difficult to maintain?
        │
        YES → Consider C++

Is it a reusable core system?
        │
        YES → C++ is often better

Does a designer need frequent iteration?
        │
        YES → Keep Blueprint exposure

This avoids converting code simply for the sake of having more C++.

Common Migration Mistakes

The biggest problems usually aren't syntax problems.

They are architecture problems.

Avoid:

converting without profiling
rewriting every Blueprint
moving presentation logic unnecessarily
creating giant C++ classes
breaking Blueprint asset references
exposing every C++ property publicly
migrating multiple large systems simultaneously
forgetting to test after reparenting

Also remember that moving a Blueprint into C++ may require updating existing references. For larger refactors, Unreal's Core Redirects can help remap references when classes or other reflected elements change.

Final Takeaway

An Unreal Blueprint to C++ conversion should be a refactor, not a declaration that Blueprint failed.

Blueprint is excellent for:

Prototype
Design
Configure
Script
Iterate

C++ becomes valuable for:

Architecture
Performance
Complex algorithms
Reusable systems
Large-scale maintenance

The strongest Unreal projects often use both.

Prototype quickly in Blueprint, identify which systems become stable or expensive, profile before optimizing, move the right foundation into C++, and expose clean interfaces back to Blueprint.

That gives programmers the control of native C++ without removing the iteration speed that makes Blueprint so valuable in the first place.

Building a WebGL Playable Ad From an Existing Game

Samcorp — Fri, 28 Aug 2026 10:30:52 GMT

How to turn an existing game into a lightweight, fast-loading WebGL playable ad without trying to squeeze the entire game into a browser ad.

Playable ads give users something normal ads cannot: a chance to experience the game before installing it.

Instead of:

Watch Ad → Click → Store → Install

the journey becomes:

Play Mini Experience → Understand Game → CTA → Install

That sounds simple, especially when the game already exists.

But building a WebGL playable ad is not the same as exporting the complete game to WebGL.

The real challenge is deciding which part of the game should survive the conversion—and then making that experience small, fast, understandable, and persuasive.

Let's walk through a practical approach.

What Is a WebGL Playable Ad?

A WebGL playable ad is a small interactive game experience that runs inside a browser or advertising environment without requiring the user to install the complete game first.

A typical structure looks like:

Ad Loads
   ↓
Instant Gameplay
   ↓
Simple Objective
   ↓
Win / Fail Moment
   ↓
Call to Action
   ↓
Install Full Game

The playable should communicate the game's core appeal within a very short interaction.

For example, if your original game is a runner:

Full Game
├── 40 levels
├── Character upgrades
├── Shop
├── Daily rewards
├── Multiple environments
└── Missions

your playable might contain only:

Playable Ad
├── One character
├── One short track
├── Basic movement
├── Coins
├── One obstacle type
└── Install CTA

That's the first important principle:

Extract the core loop. Don't port the entire game.

Step 1: Find the Most Marketable Gameplay Loop

Before touching WebGL settings, decide what users should actually experience.

Ask:

What makes the game fun within the first few seconds?

For a puzzle game, that might be:

Move → Match → Reward

For a runner:

Swipe → Avoid → Collect

For a strategy game:

Place Unit → Fight → Win

For a merge game:

Drag → Merge → Upgrade

Your playable needs one clear loop.

Avoid including:

account systems
inventories
settings
complex tutorials
multiplayer
unnecessary menus
progression systems
large maps

The playable isn't a demo of every feature.

It's a focused advertisement built around interaction.

Step 2: Create a Separate Playable Scene

Don't start deleting systems from the production game.

Create a dedicated scene:

Assets/
├── MainGame/
│
└── PlayableAd/
    ├── Scenes/
    ├── Scripts/
    ├── Art/
    └── Audio/

Then build only what the playable requires.

For example:

PlayableScene
│
├── Camera
├── Player
├── Small Environment
├── Gameplay Controller
├── UI
└── CTA Screen

This makes optimization much easier because unnecessary game systems never enter the playable build.

Teams already using Unity can also reuse gameplay logic while creating a browser-specific build pipeline. A broader Unity game development workflow can target WebGL alongside mobile and other platforms while keeping platform-specific performance requirements separate.

Step 3: Simplify the Gameplay

Suppose the original game contains:

PlayerController
InventorySystem
AchievementManager
CloudSaveManager
AnalyticsManager
MultiplayerManager
ShopManager
DailyRewardManager

The playable may only need:

PlayerController
PlayableGameManager
PlayableUI

Remove dependencies aggressively.

A playable ad should behave more like a small standalone game than a compressed production build.

Step 4: Reduce Asset Weight

Assets are often responsible for most of the downloadable size.

Start with textures.

A production asset might be:

Environment.png
2048 × 2048

For the playable, perhaps:

Environment.png
512 × 512

Also consider:

Texture atlases
Lower-resolution textures
Simpler shaders
Smaller meshes
Reduced animation frames
Compressed audio
Fewer particle effects

If the original scene uses ten environment variations, keep only the one shown in the playable.

Every asset should answer:

Does the user need this to understand why the game is fun?

If not, remove it.

Step 5: Optimize for WebGL

Once the playable scene is isolated, switch the project to WebGL.

In Unity:

File
  ↓
Build Settings
  ↓
WebGL
  ↓
Switch Platform

For production builds, focus on reducing downloadable size.

Useful settings include:

Code Optimization → Size

Strip Engine Code → Enabled

Compression → Brotli or Gzip

Development Build → Disabled

Development builds should only be used while debugging because they contain additional development information and are much larger.

Step 6: Design for Fast Startup

Playable ads compete for seconds of attention.

A user shouldn't stare at:

Loading... 13%
Loading... 27%
Loading... 44%

for several seconds before seeing anything useful.

Aim for:

Load
 ↓
Visual Feedback
 ↓
Gameplay

as quickly as possible.

This is why the first scene should contain only essential assets.

For browser-focused projects, HTML5 and WebGL game development follows the same general principle: browser experiences need lightweight assets and performance-conscious rendering because users expect interaction without an installation step.

Step 7: Make the First Interaction Obvious

Don't require a long tutorial.

Use simple visual guidance.

For example:

👆 SWIPE TO MOVE

or:

TAP TO JUMP

Then let the user interact immediately.

A good playable experience often looks like:

0–2 sec
Load + visual instruction

2–5 sec
User understands control

5–15 sec
Core gameplay

15–25 sec
Reward / challenge

25+ sec
CTA

The exact timing depends on the game and ad network, but the principle is universal:

reach gameplay quickly.

Step 8: Build a Clear CTA

Eventually, the playable needs to become an advertisement again.

After a success moment:

LEVEL COMPLETE!

Continue the Adventure

[ PLAY NOW ]

Or after failure:

SO CLOSE!

Think You Can Beat It?

[ TRY THE FULL GAME ]

The CTA should feel connected to the gameplay rather than randomly interrupting it.

Step 9: Connect JavaScript and Unity

Playable environments often require communication between Unity and the surrounding web page.

Conceptually:

Unity WebGL
     │
     ▼
JavaScript Bridge
     │
     ▼
Ad Environment
     │
     ▼
Store / CTA

For example, your game could call a JavaScript function when the user taps the CTA:

function openStore() {
    // Trigger the ad platform's
    // supported click-through behavior.
}

Keep this integration isolated.

Something like:

PlayableAdBridge.cs

should handle browser/ad-specific behavior rather than mixing it throughout gameplay code.

Step 10: Test on Real Mobile Hardware

A playable running smoothly on a developer PC proves very little.

Test it on:

Low-end Android
Mid-range Android
Recent Android
Older iPhone
Recent iPhone

Check:

Startup time
FPS
Memory
Touch input
Orientation
Audio
CTA
Browser compatibility

Also test slow network conditions.

A playable that works perfectly on office Wi-Fi may behave very differently over a weak mobile connection.

Step 11: Measure More Than Clicks

Once the playable runs correctly, track the interaction funnel.

For example:

Playable Loaded
      ↓
First Interaction
      ↓
50% Completed
      ↓
Gameplay Completed
      ↓
CTA Clicked

Useful metrics include:

Load completion rate
Interaction rate
Completion rate
Average play duration
CTA click-through rate
Install conversion rate

Now you can compare playable variants.

For example:

Version A
Immediate gameplay
CTA conversion: 6.1%

Version B
5-second tutorial
CTA conversion: 3.8%

That tells you something actionable.

Common WebGL Playable Ad Mistakes

Porting the Entire Game

The biggest mistake is treating the playable as a normal WebGL release.

Instead:

Existing Game
     ↓
Extract Core Loop
     ↓
Simplify
     ↓
Optimize
     ↓
Playable Ad

Using Production-Quality Assets Everywhere

Playable assets need to look good, but they don't always need the same resolution or complexity as the full game.

Optimize for the actual viewing environment.

Waiting Too Long for Interaction

Users should understand what to do almost immediately.

Don't waste their attention on logos, menus, and long tutorials.

Ignoring Ad-Network Requirements

Different ad platforms can impose different requirements around:

Bundle size
External requests
Audio
Orientation
Click behavior
File structure
JavaScript APIs

Always check the requirements of the network where the playable will actually run before final packaging.

Optimizing Only for File Size

A tiny playable isn't automatically a good playable.

You still need:

Fast loading
+
Smooth gameplay
+
Clear mechanics
+
Strong creative
+
Effective CTA

The goal is conversion—not winning a compression contest.

A Practical Production Workflow

A useful workflow looks like:

Existing Game
      │
      ▼
Identify Core Loop
      │
      ▼
Create Playable Scene
      │
      ▼
Remove Dependencies
      │
      ▼
Optimize Assets
      │
      ▼
Build WebGL
      │
      ▼
Integrate Ad Bridge
      │
      ▼
Test Devices
      │
      ▼
Measure Engagement
      │
      ▼
Iterate

Keeping the playable isolated from the production game makes each step easier to control.

Final Takeaway

Building a WebGL playable ad from an existing game isn't mainly an export task.

It's an editing task.

You are taking a large game experience and reducing it to the smallest interaction that still communicates:

This game is fun. I want more.

Start with the strongest gameplay loop.
Build a separate playable scene.
Remove everything the interaction doesn't need.
Compress assets and strip unused code.
Get users playing quickly.

Then finish with a clear CTA and measure what they actually do.

The best playable ad isn't the one that reproduces the most features from the original game.

It's the one that delivers the right few seconds of gameplay well enough to make users want the full experience.

Odoo API Auth in Practice: XML-RPC vs JSON-RPC

Samcorp — Fri, 28 Aug 2026 10:08:28 GMT

A practical look at Odoo XML-RPC and JSON-RPC authentication, API calls, security, and what developers should use for new integrations.

Connecting an external application to Odoo usually starts with one question:

How should the application authenticate and communicate with Odoo?

For years, two common answers have been XML-RPC and JSON-RPC.

Both can connect external applications with Odoo models, but they differ in payload format, client implementation, and developer experience.

There is also an important change for modern Odoo projects: Odoo 19 deprecates the legacy external XML-RPC and JSON-RPC endpoints in favor of the newer JSON-2 API.

So if you're working with Odoo XMLRPC JSONRPC integrations today, understanding both the existing authentication flow and the migration direction matters.

The Basic Odoo API Flow

A traditional external Odoo integration usually follows this pattern:

External Application
        │
        ▼
 Authentication
        │
        ▼
      User ID
        │
        ▼
 Odoo Object Service
        │
        ▼
     Odoo Model
        │
        ▼
      Records

Typical connection information includes:

Odoo URL
Database
Username
Password / API Key

Once authenticated, the integration can interact with models such as:

res.partner
sale.order
product.product
account.move
stock.picking

The user's normal Odoo access rights and record rules still determine what the integration can access.

For larger integrations involving CRM, eCommerce, payments, logistics, or other enterprise systems, a structured Odoo integration approach is important because authentication is only one part of reliable data synchronization.

XML-RPC Authentication

XML-RPC is one of Odoo's long-standing external API mechanisms.

In Python, the standard library already provides an XML-RPC client.

import xmlrpc.client

url = "https://example.odoo.com"
db = "example"
username = "integration@example.com"
api_key = "YOUR_API_KEY"

common = xmlrpc.client.ServerProxy(
    f"{url}/xmlrpc/2/common"
)

uid = common.authenticate(
    db,
    username,
    api_key,
    {}
)

print(uid)

If authentication succeeds, Odoo returns a user ID:

That uid is then used for model operations.

Calling an Odoo Model With XML-RPC

Create another proxy:

models = xmlrpc.client.ServerProxy(
    f"{url}/xmlrpc/2/object"
)

Now we can query contacts:

partners = models.execute_kw(
    db,
    uid,
    api_key,
    "res.partner",
    "search_read",
    [[["is_company", "=", True]]],
    {
        "fields": ["name", "email"],
        "limit": 5
    }
)

print(partners)

Conceptually, the call is:

Application
     │
     ▼
/xmlrpc/2/object
     │
     ▼
execute_kw()
     │
     ▼
res.partner.search_read()

The approach is straightforward, particularly for Python scripts and existing integrations.

What About API Keys?

For integrations, avoid hardcoding a user's main password whenever possible.

Odoo supports API keys that can replace the password in traditional RPC calls.

So instead of:

password = "user-password"

you can use:

api_key = os.environ["ODOO_API_KEY"]

and authenticate with that credential.

Keep it outside your source code:

export ODOO_API_KEY="..."

Treat the API key like a password.

Anyone who obtains it may receive the permissions associated with that Odoo user.

JSON-RPC Authentication

JSON-RPC uses JSON rather than XML to represent requests and responses.

A typical request structure looks conceptually like this:

{
  "jsonrpc": "2.0",
  "method": "call",
  "params": {
    "...": "..."
  },
  "id": 1
}

Because JSON maps naturally to JavaScript objects, JSON-RPC has historically been convenient for web-oriented integrations.

A request can be made using a normal HTTP client:

import requests

payload = {
    "jsonrpc": "2.0",
    "method": "call",
    "params": {
        "service": "common",
        "method": "authenticate",
        "args": [
            db,
            username,
            api_key,
            {}
        ]
    },
    "id": 1
}

response = requests.post(
    f"{url}/jsonrpc",
    json=payload
)

uid = response.json()["result"]

The authentication concept remains similar:

Database
   +
Username
   +
Credential
   ↓
Authenticate
   ↓
User ID

The major difference is how the RPC request is encoded and transported.

XML-RPC vs JSON-RPC

Here's the practical comparison:

Area	XML-RPC	JSON-RPC
Encoding	XML	JSON
Payload	More verbose	Generally lighter
Python support	Excellent	Excellent
JavaScript friendliness	Moderate	High
Human readability	Lower	Higher
Legacy Odoo integrations	Very common	Common
Odoo 19 external RPC status	Deprecated	Deprecated
Long-term direction	JSON-2	JSON-2

If you're maintaining an older Python integration, XML-RPC can still be perfectly understandable and functional for supported versions.

For a new long-lived integration, however, protocol choice shouldn't be made without considering Odoo's deprecation roadmap.

The Important Odoo 19 Change

This is where older Odoo API tutorials can become misleading.

Starting with Odoo 19, the external endpoints:

/xmlrpc
/xmlrpc/2
/jsonrpc

are deprecated.

Odoo's replacement is the External JSON-2 API.

The newer architecture looks more like:

POST /json/2//

with an API key supplied through the authorization header.

For example:

import requests

response = requests.post(
    f"{url}/json/2/res.partner/search_read",
    headers={
        "Authorization": f"bearer {api_key}"
    },
    json={
        "domain": [
            ["is_company", "=", True]
        ],
        "fields": [
            "name",
            "email"
        ],
        "limit": 5
    }
)

partners = response.json()

Notice what's missing:

username
password
uid
execute_kw

Authentication becomes API-key based.

The request itself identifies the model and method:

/json/2/res.partner/search_read
        │
        ├── Model  → res.partner
        │
        └── Method → search_read

This produces a cleaner HTTP integration model.

Authentication: Old RPC vs JSON-2

The difference is easier to see side by side.

XML-RPC / Legacy JSON-RPC

Database
   +
Username
   +
Password/API Key
       │
       ▼
  authenticate()
       │
       ▼
      UID
       │
       ▼
   execute_kw()

JSON-2

API Key
   │
   ▼
Authorization: bearer 
   │
   ▼
/json/2/model/method
   │
   ▼
Odoo

For new Odoo 19+ integrations, this newer model deserves serious consideration.

Use a Dedicated Integration User

Whichever API style you use, don't automatically connect integrations through an administrator account.

Create a dedicated user such as:

erp-integration-bot

and grant only the permissions required by the integration.

For example, an application that synchronizes customers may need:

Contacts → Read
Sales → Read

but probably doesn't need:

Settings → Administration
Accounting → Full Access
Users → Manage

This follows the principle of least privilege.

If an integration credential is compromised, the account's permissions determine the potential impact.

Keep Authentication Configuration Outside Code

Avoid this:

username = "admin@example.com"
api_key = "abcd1234..."

Prefer environment variables or a secrets manager:

import os

url = os.environ["ODOO_URL"]
db = os.environ["ODOO_DB"]
username = os.environ["ODOO_USER"]
api_key = os.environ["ODOO_API_KEY"]

Your repository then contains integration logic—not production credentials.

For custom modules and workflows that require API hooks or external data exchange, custom Odoo development can also separate integration logic from core ERP functionality and make future upgrades easier to manage.

Handle Authentication Failures Clearly

Don't assume authentication always succeeds.

For XML-RPC:

uid = common.authenticate(
    db,
    username,
    api_key,
    {}
)

if not uid:
    raise RuntimeError(
        "Odoo authentication failed"
    )

For HTTP-based APIs, also check:

HTTP status
Timeouts
Invalid JSON
Expired credentials
Access errors

A production integration should distinguish between:

Authentication failure
Authorization failure
Network failure
Odoo server failure
Invalid request

They require different fixes.

Which Should You Use?

The answer depends largely on the Odoo version and whether you're maintaining or creating the integration.

Existing XML-RPC integration

If it's stable and running against a supported Odoo version, you don't necessarily need an immediate rewrite.

But plan for migration.

Existing JSON-RPC integration

The same principle applies.

Keep it operational while preparing for JSON-2.

New Odoo 19+ integration

Prefer evaluating JSON-2 first.

Building a brand-new integration around an already deprecated API creates avoidable future migration work.

Older Odoo environment

XML-RPC may remain the simplest option, especially for Python automation.

Compatibility with your actual Odoo version matters more than following the newest API blindly.

A Practical Decision Guide

What are you building?
        │
        ▼
Existing integration?
   │           │
  YES          NO
   │           │
   ▼           ▼
XML/JSON RPC   Odoo 19+?
still works?    │
   │         ┌─┴─┐
  YES       YES  NO
   │         │    │
   ▼         ▼    ▼
Maintain   JSON-2  Check version
+
Plan migration

The key is not simply choosing XML or JSON.

It's choosing an API strategy that matches the lifecycle of your Odoo installation.

Security Checklist

Before shipping an Odoo API integration:

Use HTTPS.
Prefer API keys over embedding user passwords.
Store credentials outside source code.
Use a dedicated integration account.
Apply minimum required permissions.
Rotate and revoke credentials when appropriate.
Handle authentication errors explicitly.
Log failures without logging secrets.
Test record rules and access permissions.
Plan migration away from deprecated APIs.

Authentication is only secure when the entire integration respects the same trust boundary.

Final Takeaway

The Odoo XMLRPC JSONRPC comparison is no longer simply about XML versus JSON.

For existing Odoo integrations:

XML-RPC → Mature and widely used
JSON-RPC → JSON-based alternative

But for modern Odoo development:

XML-RPC
     \
      → Deprecated → JSON-2
     /
JSON-RPC

If you're maintaining an older integration, understand its authentication flow, protect API credentials, and keep permissions narrow.

If you're starting a new integration on Odoo 19 or later, evaluate JSON-2 before committing to either legacy RPC interface.

That small architecture decision today can save a much larger migration later.

Setting Up LLM Observability Without a Vendor

Samcorp — Thu, 27 Aug 2026 10:06:51 GMT

Build a vendor-neutral observability stack for LLM applications using traces, metrics, structured logs, OpenTelemetry, and your own dashboards.An LLM application fails in production.

The API returned 200 OK.

Your infrastructure dashboard is green.

CPU looks normal.

Memory looks normal.

Yet the user waited 18 seconds and received a poor answer.

What happened?

Maybe the model was slow.
Maybe the prompt became enormous.
Maybe retrieval returned irrelevant context.
Maybe an agent called the same tool four times.
Maybe a retry silently doubled the cost.

Traditional application monitoring doesn't always answer these questions.

That's where LLM observability comes in.

And you don't necessarily need to send every prompt, response, and trace to a specialized observability vendor to get started.

In this guide, we'll build a practical LLM observability setup around open telemetry concepts so that we can understand:

requests
model calls
latency
tokens
estimated cost
errors
retrieval
tool calls
end-to-end traces

while keeping control over where our telemetry goes.

What Is LLM Observability?

LLM observability is the ability to understand what happens inside an LLM-powered application by collecting and correlating telemetry.

For a basic application:

User
  │
  ▼
Application
  │
  ▼
LLM
  │
  ▼
Response

we may want to know:

Which model handled the request?

How long did generation take?

How many input tokens were consumed?

How many output tokens were generated?

Did the request fail?

How much did it approximately cost?

For an agent or RAG application, the execution path becomes more complicated:

User
 │
 ▼
Agent
 │
 ├── LLM Call
 │
 ├── Vector Search
 │
 ├── LLM Call
 │
 ├── Tool Call
 │
 ├── Tool Result
 │
 └── Final LLM Call
 │
 ▼
Response

Now observability must explain the entire execution, not just the final API request.

Monitoring vs. Observability

These terms are related but aren't identical.

Monitoring tells you:

p95 latency = 4.8 seconds
error rate = 2.1%
requests/minute = 1,200

Observability helps answer:

Why did p95 latency increase?

Perhaps traces reveal:

Retrieval          180 ms
First LLM call    1,420 ms
Tool call         3,900 ms  ← bottleneck
Second LLM call   1,610 ms

The metric tells us something is wrong.

The trace helps explain where it happened.

A useful LLM stack therefore combines:

Metrics + Traces + Logs + Evaluation Signals

Why Build It Without a Specialized Vendor?

Specialized LLM observability platforms can be useful.

But building the basic telemetry layer yourself has several advantages.

Portability

Your instrumentation doesn't have to depend on one dashboard provider.

Data control

Prompts and responses may contain:

customer information
source code
internal documents
financial information
credentials accidentally supplied by users
personal data

Keeping telemetry within infrastructure you control may simplify some privacy and security requirements.

Existing infrastructure

Your organization may already operate:

OpenTelemetry
Prometheus
Grafana
Jaeger
Tempo
Loki
Elasticsearch

Adding LLM signals to the same stack may be easier than creating another monitoring silo.

Understanding

Building the first version yourself forces you to decide which signals actually matter.

That's valuable even if you eventually adopt a specialized platform.

The Architecture We're Building

Our initial architecture will look like this:

                    LLM Application
                           │
                           │ telemetry
                           ▼
                 OpenTelemetry SDK
                           │
                           ▼
                  OTel Collector
                    /      |      \
                   /       |       \
                  ▼        ▼        ▼
              Traces    Metrics    Logs
                 │         │         │
                 ▼         ▼         ▼
               Tempo   Prometheus   Loki
                  \        |        /
                   \       |       /
                    ▼      ▼      ▼
                       Grafana

You don't have to use these exact backends.

That's one of the advantages of the architecture.

The application emits standardized telemetry.

The backend is replaceable.

Step 1: Decide What You Need to Observe

Don't start by logging everything.

Start with questions.

For example:

Why is this request slow?

Which model is consuming the most tokens?

Which prompt version causes the most failures?

How often are tools failing?

How much does one successful request cost?

Which retrieval step adds the most latency?

Those questions define your telemetry requirements.

For a first version, collect:

Request ID
Trace ID
Model
Operation
Latency
Input tokens
Output tokens
Finish reason
Status
Error type
Prompt version
Application version
Environment

For agents, add:

Tool name
Tool duration
Tool result status
Agent turns
Retry count

For RAG:

Retriever
Retrieval duration
Number of documents
Embedding model
Top-k
Context size

That's already enough to investigate many production problems.

Step 2: Don't Log Full Prompts by Default

This deserves its own step.

The easiest observability implementation is:

logger.info(prompt)
logger.info(response)

It's also potentially dangerous.

Prompts may contain:

PII
customer records
internal documents
source code
access tokens
medical information
financial information

Instead, begin with metadata:

{
  "trace_id": "8c94...",
  "model": "example-model",
  "prompt_version": "support-v12",
  "input_tokens": 812,
  "output_tokens": 184,
  "duration_ms": 1420,
  "status": "ok"
}

If you later decide that prompt capture is necessary, make it an explicit feature with:

redaction
sampling
access controls
retention policies
encryption
audit logging

Observability shouldn't become a new data-leakage system.

Step 3: Start With OpenTelemetry

OpenTelemetry gives us a vendor-neutral model for telemetry.

Instead of wiring our application directly to one backend:

Application
    │
    ▼
Vendor-specific SDK
    │
    ▼
Vendor

we can use:

Application
    │
    ▼
OpenTelemetry
    │
    ▼
OTLP
    │
    ├── Backend A
    ├── Backend B
    └── Self-hosted stack

This separation is one of the most important design choices in a vendor-neutral observability system.

Step 4: Instrument the Application

Let's use Python for a minimal example.

Install OpenTelemetry packages:

pip install \
  opentelemetry-api \
  opentelemetry-sdk \
  opentelemetry-exporter-otlp

Create a tracer:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter
)

resource = Resource.create({
    "service.name": "llm-api",
    "service.version": "1.0.0",
    "deployment.environment": "production"
})

provider = TracerProvider(resource=resource)

provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint="http://otel-collector:4317",
            insecure=True
        )
    )
)

trace.set_tracer_provider(provider)

tracer = trace.get_tracer("llm-observability")

Now the application can create spans.

Step 5: Trace an LLM Call

Wrap model execution in a span.

def call_llm(client, model, messages):

    with tracer.start_as_current_span(
        "llm.generate"
    ) as span:

        span.set_attribute(
            "gen_ai.request.model",
            model
        )

        response = client.generate(
            model=model,
            messages=messages
        )

        span.set_attribute(
            "gen_ai.usage.input_tokens",
            response.usage.input_tokens
        )

        span.set_attribute(
            "gen_ai.usage.output_tokens",
            response.usage.output_tokens
        )

        return response

Now a model request isn't just:

POST /chat

It becomes part of a distributed trace.

Step 6: Trace the Whole Request

The model call should normally be a child of the user request.

def handle_request(question):

    with tracer.start_as_current_span(
        "answer_question"
    ):

        context = retrieve_context(question)

        answer = call_llm(
            client,
            MODEL,
            build_messages(
                question,
                context
            )
        )

        return answer

The resulting trace might look like:

answer_question                 2.83s
│
├── retrieve_context            0.31s
│
└── llm.generate                2.47s

This is where tracing becomes much more useful than disconnected logs.

You can see where the time went.

Step 7: Instrument Retrieval

For a RAG system:

def retrieve_context(query):

    with tracer.start_as_current_span(
        "rag.retrieve"
    ) as span:

        results = vector_store.search(
            query,
            top_k=5
        )

        span.set_attribute(
            "rag.top_k",
            5
        )

        span.set_attribute(
            "rag.documents.returned",
            len(results)
        )

        return results

Now the trace becomes:

answer_question
│
├── rag.retrieve
│
└── llm.generate

Later you might add:

embedding generation
reranking
document filtering
context construction

as separate spans.

Step 8: Instrument Tool Calls

Agent applications need another layer.

Suppose the model calls:

get_order_status

Trace it separately:

def execute_tool(name, arguments):

    with tracer.start_as_current_span(
        f"tool.{name}"
    ) as span:

        span.set_attribute(
            "tool.name",
            name
        )

        try:

            result = tool_registry[name](
                **arguments
            )

            span.set_attribute(
                "tool.status",
                "success"
            )

            return result

        except Exception as error:

            span.record_exception(error)

            span.set_attribute(
                "tool.status",
                "error"
            )

            raise

Now an agent trace might show:

agent.run                         7.4s
│
├── llm.generate                 1.8s
│
├── tool.get_order_status        3.2s
│
└── llm.generate                 2.1s

Immediately we know the model wasn't responsible for most of the delay.

The internal tool was.

Step 9: Add Metrics

Traces explain individual requests.

Metrics reveal patterns across thousands of them.

At minimum, track:

LLM request count
LLM request duration
Input tokens
Output tokens
Errors
Tool duration
Tool errors

Useful dashboards might include:

Requests/min
p50 latency
p95 latency
p99 latency
Token usage/min
Average tokens/request
Errors by model
Latency by model
Tool failure rate

For production AI systems, observability becomes part of the broader deployment lifecycle rather than something added only after incidents. This is why a mature generative AI development process typically combines testing, deployment, monitoring, guardrails, and ongoing optimization.

Step 10: Estimate Cost Yourself

If you know a model's current token pricing, cost calculation is straightforward.

Conceptually:

def estimate_cost(
    input_tokens,
    output_tokens,
    input_price,
    output_price
):
    return (
        input_tokens / 1_000_000
        * input_price
    ) + (
        output_tokens / 1_000_000
        * output_price
    )

Then attach the result:

span.set_attribute(
    "llm.estimated_cost_usd",
    cost
)

You can now answer:

What does one request cost?

Which endpoint costs the most?

Which model consumes the most budget?

Did the latest prompt increase token usage?

Keep pricing configuration outside application code so it can be updated without changing instrumentation.

Step 11: Track Prompt Versions, Not Necessarily Prompt Text

Suppose production currently uses:

support-agent-v17

Record:

span.set_attribute(
    "app.prompt.version",
    "support-agent-v17"
)

rather than storing the entire system prompt on every request.

Now you can compare:

Prompt v15
p95 latency: 2.8s
avg input: 720 tokens

Prompt v16
p95 latency: 3.0s
avg input: 890 tokens

Prompt v17
p95 latency: 4.4s
avg input: 1,620 tokens

Suddenly a latency regression has an obvious clue.

Prompt versioning gives you useful correlation without automatically storing sensitive content.

Step 12: Add Structured Logs

You still need logs.

Just make them structured.

Instead of:

LLM failed!!!

write:

{
  "level": "error",
  "event": "llm_request_failed",
  "trace_id": "83a21...",
  "model": "example-model",
  "prompt_version": "support-v17",
  "error_type": "timeout",
  "retry_count": 2
}

The most important field is often:

trace_id

It connects logs with traces.

A useful debugging workflow becomes:

Alert
  ↓
Metric
  ↓
Trace
  ↓
Span
  ↓
Correlated logs

Step 13: Deploy an OpenTelemetry Collector

Don't have every application export directly to storage.

Put a collector between them.

Applications
     │
     │ OTLP
     ▼
+---------------------+
| OpenTelemetry       |
| Collector           |
+---------------------+
     │
     ├── traces
     ├── metrics
     └── logs

The collector gives you a central place for:

batching
filtering
sampling
redaction
routing
retries
export

Your applications only need to know where the collector lives.

Step 14: Configure the Collector

A simplified configuration might look like:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:

    traces:
      receivers:
        - otlp

      processors:
        - batch

      exporters:
        - otlp/tempo

You can later add separate pipelines for:

metrics
logs

and additional processors for filtering or sampling.

Step 15: Build a Local Stack

A useful self-hosted stack is:

OpenTelemetry Collector
        │
        ├── Tempo      → traces
        ├── Prometheus → metrics
        └── Loki       → logs
                       │
                       ▼
                    Grafana

Another option is:

OpenTelemetry
      │
      ▼
Jaeger

for a simpler tracing-focused setup.

The important part isn't the exact backend.

It's keeping instrumentation separate from storage.

Step 16: Create Your First Dashboard

Don't build 40 charts.

Start with a small operational dashboard.

Traffic

Requests/minute
Requests by model

Latency

p50
p95
p99

Tokens

Input tokens/min
Output tokens/min
Tokens/request

Reliability

Error rate
Timeout rate
Retry rate

Cost

Estimated cost/hour
Estimated cost/request
Cost by model

Agent Tools

Tool calls
Tool latency
Tool failures

That's enough to answer many operational questions.

Step 17: Add Alerts

Dashboards require someone to look at them.

Alerts tell you when something changes.

For example:

p95 latency > 8 seconds
for 10 minutes

or:

LLM error rate > 5%

or:

Tool failure rate > 10%

Token anomalies can also be useful:

Average input tokens increased
more than 40% from baseline

This can detect:

prompt changes
retrieval explosions
conversation-history growth
agent loops

before the cost increase becomes obvious on the monthly bill.

Step 18: Add Sampling Before Volume Explodes

Keeping every trace forever is rarely necessary.

A simple strategy might be:

Successful request → sample 5%

Error → keep 100%

Very slow request → keep 100%

Critical workflow → keep 100%

This is much more useful than blindly sampling every request at the same rate.

The interesting traces are often the unusual ones.

Step 19: Redact at the Collector

Suppose telemetry accidentally contains:

user.email
authorization.header
customer.phone

You don't necessarily want those values reaching storage.

A useful architecture is:

Application
     │
     ▼
OTel Collector
     │
     ├── Remove sensitive attributes
     ├── Redact values
     ├── Sample
     └── Route
     │
     ▼
Telemetry Storage

This creates a centralized privacy control point.

Still, prevention at the application layer is better than relying exclusively on downstream redaction.

Step 20: Observe RAG as a Pipeline

A common mistake is treating RAG as one LLM request.

It's actually a pipeline:

Question
   │
   ▼
Embedding
   │
   ▼
Vector Search
   │
   ▼
Reranking
   │
   ▼
Context Construction
   │
   ▼
LLM

Trace these separately:

rag.request
│
├── embedding.create
├── vector.search
├── reranker.rank
├── context.build
└── llm.generate

Now when RAG quality deteriorates, you can investigate the individual stages.

Step 21: Observe Agents as Trees

Agent executions are even more interesting.

A trace could look like:

agent.run
│
├── llm.generate
│
├── tool.search_customer
│
├── llm.generate
│
├── tool.get_orders
│
├── llm.generate
│
└── response

From that single trace you can calculate:

number of LLM calls
number of tool calls
total tokens
total latency
tool latency
retries
estimated cost

This makes traces especially valuable for debugging agentic systems.

Step 22: Detect Agent Loops

Imagine this trace:

agent.run
│
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── llm.generate
├── tool.search
├── ...

Your application is probably stuck.

Create metrics such as:

agent_turns
tool_calls_per_run
llm_calls_per_run

Then alert on abnormal values.

For example:

agent_turns > 12

or:

same tool called > 5 times

Observability can reveal problems that ordinary API monitoring completely misses.

Step 23: Add Quality Signals

Infrastructure telemetry answers:

Was it fast?

Did it fail?

How many tokens did it use?

But an LLM can be:

fast
cheap
error-free

and still give a terrible answer.

So eventually add evaluation signals:

groundedness
correctness
user feedback
task completion
hallucination rate
retrieval quality

Then correlate them with traces.

For example:

Trace ID: abc123

Latency:       2.8s
Cost:          $0.009
Tokens:        1,240
Groundedness:  0.91
User rating:   positive

Now you're observing both system performance and AI behavior.

Step 24: Define LLM SLOs

Traditional systems have SLOs.

LLM systems should too.

For example:

Reliability

99.5% successful model calls

Latency

95% of requests complete < 6 seconds

Cost

95% of requests cost < $0.03

Agent behavior

99% complete within 8 turns

Quality

groundedness score > agreed threshold

The exact values depend on the application.

The important part is defining acceptable behavior before an incident.

For larger production AI systems, this operating layer should be planned alongside model deployment, security, evaluation, and lifecycle management. SDLC Corp's AI development services similarly describe deployment and monitoring as part of the broader AI lifecycle rather than treating monitoring as a separate afterthought.

Step 25: Build a Debugging Workflow

The biggest benefit of observability isn't the dashboard.

It's reducing the time between:

Something is wrong.

and:

We know why.

A useful workflow looks like:

Alert
  │
  ▼
Find affected metric
  │
  ▼
Filter by model/version
  │
  ▼
Open representative trace
  │
  ▼
Find slow/failing span
  │
  ▼
Inspect correlated logs
  │
  ▼
Identify root cause

For example:

ALERT:
p95 latency > 8s

        ↓

FILTER:
prompt_version = support-v24

        ↓

TRACE:
rag.retrieve = 5.4s

        ↓

ROOT CAUSE:
vector search latency regression

That's observability doing useful engineering work.

What Should You Avoid Collecting?

Your telemetry policy should explicitly define what should not be collected.

Usually that includes:

Passwords
API keys
Authorization headers
Access tokens
Raw secrets
Payment details
Unnecessary PII
Sensitive documents

For prompt and completion content, make a deliberate risk-based decision.

Don't enable content capture simply because the instrumentation supports it.

A Practical Production Architecture

A mature setup could eventually look like:

                  LLM Application
                         │
                         ▼
               OpenTelemetry SDK
                         │
                         ▼
                 OTel Collector
                         │
             ┌───────────┼───────────┐
             │           │           │
             ▼           ▼           ▼
          Metrics      Traces       Logs
             │           │           │
             ▼           ▼           ▼
        Prometheus     Tempo        Loki
             │           │           │
             └───────────┼───────────┘
                         │
                         ▼
                      Grafana
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Alerts      Dashboards   Debugging

Then add:

Evaluation results
User feedback
Cost data
Release metadata
Prompt versions
Model versions

as your system matures.

LLM Observability Setup Checklist

Before calling the setup complete, verify:

Instrumentation

End-to-end requests have trace IDs
LLM calls are individual spans
Retrieval operations are traced
Tool calls are traced
Errors are recorded

Metrics

Request volume
Latency
Input tokens
Output tokens
Error rate
Retry rate
Estimated cost

Metadata

Model
Prompt version
Application version
Environment

Privacy

Secrets are excluded
PII policy exists
Prompt capture is intentional
Retention is defined
Telemetry access is restricted

Operations

Dashboard exists
Alerts exist
Sampling is configured
Engineers know how to find a trace from an incident

Common LLM Observability Mistakes

Logging Everything

More data isn't automatically more observability.

Collect information that helps answer operational questions.

Capturing Prompts Without a Privacy Plan

Prompt logging can expose sensitive information.

Start metadata-first.

Tracking Only Latency

A fast LLM request can still be expensive or incorrect.

Track:

latency + tokens + cost + errors + quality

Tracking Only the LLM

For RAG and agents, the model is only one part of the system.

Trace:

retrieval
tools
databases
external APIs
model calls

together.

Ignoring Versions

Always record enough information to correlate regressions with changes:

model version
prompt version
application version
retrieval configuration

Building Vendor-Specific Instrumentation Too Early

If possible, instrument around open telemetry standards and adapt at the export layer.

That keeps future backend changes much easier.

What You Actually Need to Start

You don't need a huge LLMOps platform on day one.

Start with:

OpenTelemetry SDK
        ↓
OTel Collector
        ↓
Traces + Metrics
        ↓
Dashboard

Instrument:

LLM calls
retrieval
tools

and record:

latency
tokens
errors
model
prompt version
trace ID

That's already enough to answer a surprising number of production questions.

Add complexity only when a real problem requires it.

Final Takeaway

A good LLM observability setup isn't about collecting every possible detail from every prompt.

It's about being able to answer:

What happened, where did it happen, why did it happen, and what changed?

Build around open telemetry rather than a specific dashboard.

Start with metadata rather than sensitive content.

Trace the entire application rather than only the model call.

Measure tokens and cost alongside traditional reliability metrics.

Correlate prompts, models, retrieval, and application releases through version metadata.

Then gradually connect operational telemetry with evaluation results and user feedback.

The final goal isn't a beautiful dashboard.

It's reaching the point where an LLM application behaves strangely in production and your team can explain why without guessing.

Implementing MCP Servers for Internal Tools: A Practical Developer Guide

Samcorp — Thu, 27 Aug 2026 09:14:20 GMT

Learn how to design and implement an MCP server that safely connects AI applications to internal APIs, databases, services, and enterprise workflows.

Giving an AI assistant access to company systems sounds simple:

LLM → Internal API → Result

In practice, it quickly becomes messy.

One application needs access to customer records. Another needs Jira-style project data. A developer assistant needs deployment information. An operations agent needs monitoring tools.

Before long, every AI application has its own integrations:

AI Assistant ───────→ CRM API
Developer Agent ────→ Deployment API
Support Agent ──────→ Ticket System
Analytics Agent ────→ Internal Database

Each integration requires its own schemas, authentication logic, error handling, permissions, and documentation.

Model Context Protocol (MCP) gives us another approach.

Instead of teaching every AI application how every internal system works, we can put an MCP server between them.

AI Application
       │
       │ MCP
       ▼
+------------------+
|    MCP Server    |
+------------------+
   │      │      │
   ▼      ▼      ▼
 CRM     DB     Internal API

The server exposes carefully controlled capabilities that an MCP-compatible application can discover and use.

In this tutorial, we'll walk through MCP server implementation for internal tools, starting with architecture and ending with security, testing, and production deployment.

What Is an MCP Server?

Model Context Protocol is an open protocol for connecting AI applications with external systems and context.

An MCP server sits on the system side of that relationship.

Instead of exposing an entire internal application directly to an AI model, the server exposes specific MCP capabilities.

The three primitives you'll encounter most often are:

Tools

Tools allow the AI application to perform actions or calculations.

Examples:

get_customer
search_tickets
create_support_ticket
check_deployment
restart_service
generate_report

Resources

Resources expose information that can be read as context.

Examples:

internal documentation
configuration
database-backed records
service information
project metadata
runbooks

Prompts

Prompts expose reusable interaction templates.

For example:

summarize_incident
review_deployment
prepare_customer_report

The important architectural idea is that the model doesn't need to understand your internal API implementation.

It only needs to understand the MCP interface you expose.

Why MCP Makes Sense for Internal Tools

Imagine your organization has:

CRM
ERP
Git repositories
CI/CD platform
Analytics database
Knowledge base
Monitoring system
Support platform
Internal REST APIs

Without a common integration layer, every AI application may need separate connectors.

With MCP:

                    ┌───────────────┐
                    │ AI Assistant  │
                    └───────┬───────┘
                            │
                    ┌───────▼───────┐
                    │  MCP Client   │
                    └───────┬───────┘
                            │
                           MCP
                            │
                    ┌───────▼───────┐
                    │  MCP Server   │
                    └───────┬───────┘
                            │
           ┌────────────────┼────────────────┐
           │                │                │
           ▼                ▼                ▼
          CRM            Database        DevOps API

The MCP server becomes a controlled interface between AI applications and business infrastructure.

This pattern fits particularly well with broader enterprise AI development where LLMs need to interact with existing internal platforms while organizations still need access control, governance, and secure integration.

Step 1: Decide What the AI Actually Needs

A common mistake is starting with:

"Let's expose our internal API through MCP."

That's too broad.

Start with specific tasks instead.

Suppose employees repeatedly ask:

What's the status of order 58421?

Find the customer associated with this email.

Is checkout-api deployed in production?

Show me unresolved P1 incidents.

These translate naturally into tools:

get_order_status
find_customer
get_deployment_status
list_critical_incidents

This gives us a much safer boundary.

Instead of exposing:

execute_database_query
call_internal_api
run_shell_command

we expose business-level operations:

get_order
find_customer
check_service_health

That difference is critical.

The first group gives an AI broad execution capabilities.

The second gives it narrowly defined actions.

Step 2: Choose Your Transport

The transport determines how an MCP client communicates with the server.

For many local developer integrations, stdio is the simplest choice.

Conceptually:

MCP Host
   │
   ├── starts server process
   │
   ▼
MCP Server
stdin  ← requests
stdout → protocol responses

This works well when the host launches the server locally.

For a centrally hosted internal MCP service used by multiple clients, an HTTP-based deployment is usually more appropriate.

Conceptually:

Developer Laptop
       │
       │ HTTPS
       ▼
Company MCP Endpoint
       │
       ├── CRM
       ├── Database
       └── Internal APIs

Keep transport decisions separate from your business logic so the same tools can eventually be exposed through a different deployment model.

Step 3: Create the Project

We'll use TypeScript.

A simple structure might be:

internal-mcp/
│
├── src/
│   ├── index.ts
│   ├── tools/
│   │   ├── customer.ts
│   │   ├── deployments.ts
│   │   └── incidents.ts
│   │
│   ├── services/
│   │   ├── crm.ts
│   │   └── devops.ts
│   │
│   └── security/
│       └── permissions.ts
│
├── package.json
└── tsconfig.json

Install the current MCP server package and Zod:

npm install @modelcontextprotocol/server zod

During development, you'll also want your normal TypeScript tooling.

Keeping tools separate from service integrations will become important as the server grows.

Step 4: Create the MCP Server

Start with a server factory.

import { McpServer } from "@modelcontextprotocol/server";
import { serveStdio } from "@modelcontextprotocol/server/stdio";
import * as z from "zod/v4";

function createServer() {
  const server = new McpServer({
    name: "internal-tools",
    version: "1.0.0"
  });

  return server;
}

void serveStdio(createServer);

This gives us the foundation.

Right now the server isn't particularly useful because it doesn't expose anything.

Let's fix that.

Step 5: Implement Your First Tool

Suppose we have an internal deployment service.

We want an AI assistant to answer:

"What's currently deployed for checkout-api?"

Our tool should represent that exact business operation.

server.registerTool(
  "get-deployment-status",
  {
    title: "Get Deployment Status",

    description:
      "Get the current deployment status for an internal service.",

    inputSchema: {
      service: z.string().min(1),
      environment: z.enum([
        "development",
        "staging",
        "production"
      ])
    }
  },

  async ({ service, environment }) => {
    const deployment = await deploymentService.getStatus(
      service,
      environment
    );

    return {
      content: [
        {
          type: "text",
          text: JSON.stringify(deployment)
        }
      ]
    };
  }
);

An MCP client can now discover that capability and understand its input requirements.

For example:

{
  "service": "checkout-api",
  "environment": "production"
}

could return:

{
  "service": "checkout-api",
  "environment": "production",
  "version": "v2.18.4",
  "status": "healthy"
}

Step 6: Keep MCP Logic Thin

Don't put all your business logic inside tool handlers.

Avoid turning this:

server.registerTool(...)

into hundreds of lines of API calls, database queries, authorization checks, transformations, and retries.

Instead, keep a service layer:

class DeploymentService {
  async getStatus(
    service: string,
    environment: string
  ) {
    // Call internal deployment platform.

    return {
      service,
      environment,
      version: "v2.18.4",
      status: "healthy"
    };
  }
}

Then the MCP layer does only a few things:

Validate input
      ↓
Check permission
      ↓
Call service
      ↓
Transform result
      ↓
Return MCP response

Your architecture becomes:

MCP Protocol
     │
     ▼
Tool Handler
     │
     ▼
Service Layer
     │
     ▼
Internal System

This separation also makes unit testing much easier.

Step 7: Add a Customer Lookup Tool

Now let's connect another internal system.

server.registerTool(
  "find-customer",
  {
    title: "Find Customer",

    description:
      "Find a customer using their company email address.",

    inputSchema: {
      email: z.string().email()
    }
  },

  async ({ email }) => {
    const customer = await crmService.findByEmail(email);

    if (!customer) {
      return {
        content: [
          {
            type: "text",
            text: "Customer not found."
          }
        ]
      };
    }

    return {
      content: [
        {
          type: "text",
          text: JSON.stringify({
            id: customer.id,
            name: customer.name,
            accountStatus: customer.accountStatus
          })
        }
      ]
    };
  }
);

Notice something important.

Our CRM record might contain:

Customer ID
Name
Email
Phone
Billing address
Internal notes
Payment information
Account status
Support history

But the MCP tool returns only:

ID
Name
Account status

That is intentional.

Step 8: Apply Data Minimization

One of the biggest mistakes in internal AI integrations is returning everything simply because the backend API provides it.

Don't do this:

return entireCustomerObject;

Instead:

return {
  id: customer.id,
  name: customer.name,
  accountStatus: customer.accountStatus
};

The question should always be:

What is the minimum information necessary for this tool to complete its job?

This reduces:

accidental data exposure
unnecessary model context
token usage
privacy risk
confusing responses

MCP does not remove your responsibility to design safe application boundaries.

It gives you a standardized interface through which to enforce them.

Step 9: Use Resources for Read-Only Context

Not everything needs to be a tool.

Suppose your engineering assistant needs access to an internal deployment policy.

A resource is often a better abstraction.

Conceptually:

resource:
internal://policies/deployment

could return:

# Production Deployment Policy

- Production deployments require approval.
- Deployments must pass integration tests.
- Rollback procedures must be documented.

Think about the distinction like this:

Need information?
      │
      └── Resource

Need to perform an operation?
      │
      └── Tool

Need a reusable interaction workflow?
      │
      └── Prompt

This produces a cleaner MCP interface.

Step 10: Design Tool Names for Models, Not APIs

Your internal API might have an endpoint named:

GET /api/v4/env/service/current

Don't expose that mental model.

Expose:

get-deployment-status

Similarly:

POST /crm/v3/search/entity

becomes:

find-customer

Tool names and descriptions are part of the interface the model reasons about.

Make them:

explicit
narrow
predictable
action-oriented

Avoid ambiguous tools such as:

execute
process
manage
run
perform

Prefer:

get-order-status
create-support-ticket
list-open-incidents
get-deployment-status

Step 11: Write Precise Tool Descriptions

Compare these:

Gets service information.

and:

Returns the currently deployed version and health
status for an internal service in development,
staging, or production.

The second description gives the model much more information about when the tool is appropriate.

Descriptions should answer:

What does this tool do?
When should it be used?
What does it return?
What important limitations exist?

Don't hide important business rules only inside the implementation.

Step 12: Add Authorization Before Real Actions

Internal tools often have very different risk levels.

Reading service status is not equivalent to restarting production.

We can classify operations:

Tool	Risk
`get-deployment-status`	Low
`search-customer`	Medium
`create-ticket`	Medium
`deploy-service`	High
`restart-production-service`	Critical

Your authorization layer should reflect that.

Conceptually:

async function authorize(
  user: User,
  action: string
) {
  const allowed = await permissionService.can(
    user.id,
    action
  );

  if (!allowed) {
    throw new Error("Permission denied");
  }
}

Then:

await authorize(
  currentUser,
  "deployment:read"
);

For sensitive operations, you may need additional approval or confirmation before the action is executed.

The important rule is:

Never treat access to the MCP server itself as authorization to every tool it exposes.

Step 13: Separate Read and Write Operations

A useful first production milestone is a read-only MCP server.

Start with:

get_customer
get_order
list_incidents
read_documentation
check_deployment
get_service_health

Then introduce writes gradually:

create_ticket
update_customer
trigger_build
restart_service
deploy_application

Write operations deserve stricter controls because their failures change real systems.

A useful pattern is:

Read
 ↓
Prepare action
 ↓
Validate
 ↓
Authorize
 ↓
Confirm when necessary
 ↓
Execute
 ↓
Audit

Step 14: Never Give the Model Raw Database Access

This is tempting:

execute_sql(query)

It's also an unnecessarily large security boundary.

A better approach is:

get_customer_orders(customer_id)

instead of:

SELECT *
FROM orders
WHERE customer_id = ...

Likewise:

get_monthly_revenue(month)

is safer than:

execute_analytics_sql

The MCP server should act as a controlled application layer—not simply turn the model into a database administrator.

Step 15: Treat Tool Arguments as Untrusted Input

A model-generated tool call is still input.

Validate it exactly as you would validate an external API request.

Zod schemas help here:

inputSchema: {
  service: z
    .string()
    .min(1)
    .max(100),

  environment: z.enum([
    "development",
    "staging",
    "production"
  ])
}

For identifiers, define formats:

ticketId: z
  .string()
  .regex(/^TICKET-[0-9]+$/)

Don't rely on prompts such as:

Please only provide valid service names.

Prompts are not validation.

Code is.

Step 16: Protect Against Tool Chaining Risks

Consider this sequence:

1. Read internal document
2. Extract instruction from document
3. Call another tool
4. Modify production system

If untrusted content influences tool selection, an AI agent could potentially execute actions you never intended.

This is one reason internal MCP security isn't simply:

Authentication = solved

You also need to consider:

Authentication
Authorization
Input validation
Output filtering
Least privilege
Confirmation
Auditability
Tool interactions
Data classification

The MCP layer becomes part of your security architecture.

Step 17: Add Audit Logging

For production internal tools, you should be able to reconstruct what happened.

A useful audit event might contain:

{
  "timestamp": "2026-08-27T08:30:00Z",
  "actor": "user-1842",
  "tool": "get-deployment-status",
  "arguments": {
    "service": "checkout-api",
    "environment": "production"
  },
  "result": "success",
  "duration_ms": 183
}

For write operations, consider recording:

who initiated the request
which tool was invoked
which target was affected
authorization result
approval/confirmation
execution outcome
timestamp
request/correlation ID

Be careful not to create a second privacy problem by logging secrets or sensitive payloads.

Step 18: Handle Errors Deliberately

Internal systems fail.

Your CRM may timeout.

Your deployment API may be unavailable.

A requested customer may not exist.

Don't convert everything into:

Internal server error

Instead, create predictable error categories:

NOT_FOUND
PERMISSION_DENIED
INVALID_INPUT
UPSTREAM_TIMEOUT
RATE_LIMITED
SERVICE_UNAVAILABLE

Then map internal errors into safe messages.

For example:

try {
  const result = await deploymentService.getStatus(
    service,
    environment
  );

  return toMcpResult(result);
} catch (error) {
  logger.error({
    error,
    service,
    environment
  });

  return {
    isError: true,
    content: [
      {
        type: "text",
        text: "Deployment status is temporarily unavailable."
      }
    ]
  };
}

The user receives useful information without exposing stack traces, credentials, internal hostnames, or infrastructure details.

Step 19: Add Timeouts

Never assume an internal service will respond.

Conceptually:

const result = await withTimeout(
  deploymentService.getStatus(
    service,
    environment
  ),
  5000
);

Without timeouts, a single unhealthy dependency can leave tool execution hanging.

For production integrations, also consider:

timeouts
retry policies
circuit breakers
rate limits
connection limits
bulkheads
fallback behavior

Step 20: Keep Credentials Out of Tool Arguments

Avoid tool schemas like:

{
  "apiKey": "...",
  "customerId": "..."
}

Credentials should come from your server-side environment or identity infrastructure.

The flow should be:

MCP Client
    │
    │ authorized request
    ▼
MCP Server
    │
    ├── identity
    ├── permissions
    └── server-side credentials
             │
             ▼
       Internal Service

not:

Model → Secret → Tool → Internal API

The model should receive as few secrets as possible—ideally none.

Step 21: Design Around Least Privilege

Suppose an AI coding assistant needs to inspect deployment state.

It probably needs:

deployment:read

It probably does not need:

deployment:create
deployment:delete
production:restart
secret:read

The same principle applies to backend credentials.

If the MCP server only reads tickets, its service account shouldn't have permission to delete them.

Use least privilege at multiple layers:

User
 ↓
MCP tool permissions
 ↓
Service identity
 ↓
Internal API permissions
 ↓
Database permissions

If one layer fails, another still limits the blast radius.

Step 22: Test the MCP Server

For local development, the MCP Inspector is extremely useful.

You can launch an stdio server through the Inspector:

npx @modelcontextprotocol/inspector npx tsx src/index.ts

Then inspect available tools and execute them manually.

Test at least:

valid requests
invalid arguments
missing records
unauthorized requests
upstream failures
timeouts
malformed upstream responses
sensitive-data filtering
write confirmation paths

Don't test only:

Does the tool work?

Also test:

Can the tool do something it shouldn't?

Step 23: Be Careful With stdio Logging

When your server runs over stdio, standard output is part of the protocol channel.

So this can cause problems:

console.log("Server started");

Use stderr for development logging instead:

console.error("Internal MCP server started");

And in production, route structured logs through an appropriate logging pipeline.

Small transport details like this can save a surprising amount of debugging time.

Step 24: Move Toward a Shared Internal MCP Service

A local stdio server is excellent for development.

Organizations may eventually want something more centralized:

                 ┌─────────────────┐
                 │ AI Application  │
                 └────────┬────────┘
                          │
                 ┌────────▼────────┐
                 │ Identity Layer  │
                 └────────┬────────┘
                          │
                 ┌────────▼────────┐
                 │   MCP Gateway   │
                 └────────┬────────┘
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
          CRM MCP      DevOps MCP    Data MCP
             │            │            │
             ▼            ▼            ▼
            CRM         CI/CD       Analytics

Now you can apply centralized:

authentication
authorization
observability
rate limiting
auditing
network policies
secret management

while keeping individual MCP servers focused on specific domains.

This is also where MCP implementation becomes part of a larger AI integration strategy. Organizations building AI services integrated with existing business systems need to think beyond the model itself and design secure integration, deployment, monitoring, access-control, and governance layers around the AI system.

Step 25: Avoid Building One Giant MCP Server

It can be tempting to create:

company-mcp-server

containing 300 tools.

That quickly becomes difficult to reason about.

A domain-oriented design is often cleaner:

crm-mcp
├── find_customer
├── get_account
└── list_customer_tickets

devops-mcp
├── get_deployment
├── list_builds
└── get_service_health

support-mcp
├── search_tickets
├── get_ticket
└── create_ticket

knowledge-mcp
├── search_docs
└── read_document

Benefits include:

smaller permission boundaries
clearer ownership
simpler deployments
easier testing
better tool discovery
reduced blast radius

Your organization can then decide which AI applications receive access to which servers.

A Production-Oriented Architecture

Once the prototype works, the architecture might evolve into:

                  AI Application
                         │
                         ▼
                 MCP Client / Host
                         │
                         ▼
              ┌─────────────────────┐
              │ Authentication      │
              ├─────────────────────┤
              │ Authorization       │
              ├─────────────────────┤
              │ MCP Server          │
              ├─────────────────────┤
              │ Tool Validation     │
              ├─────────────────────┤
              │ Service Layer       │
              ├─────────────────────┤
              │ Audit / Metrics     │
              └──────────┬──────────┘
                         │
              ┌──────────┼──────────┐
              ▼          ▼          ▼
             CRM      Internal DB   DevOps

Each layer has a specific job.

The MCP protocol defines the interface.

Your application still defines the trust boundary.

MCP Server Implementation Checklist

Before exposing an internal MCP server, ask:

Interface

Are tools narrowly scoped?
Are names understandable?
Are descriptions precise?
Are schemas strict?
Are read and write operations clearly separated?

Security

Is authentication enforced?
Is authorization checked per operation?
Are permissions least-privileged?
Are secrets kept outside model context?
Are dangerous operations protected?
Are sensitive outputs filtered?

Reliability

Are upstream calls timed out?
Are failures handled predictably?
Are retries bounded?
Are dependencies observable?

Governance

Are important actions audited?
Can you identify the requesting actor?
Can security teams investigate an incident?
Is sensitive information excluded from logs?

Testing

Have invalid inputs been tested?
Have unauthorized actions been tested?
Have upstream failures been simulated?
Have write operations been tested safely?

If several of those answers are "no," the server probably isn't ready for production.

Common MCP Implementation Mistakes

1. Exposing Generic API Tools

Avoid:

call_api
execute_sql
run_command

Prefer narrow business operations.

2. Treating the Model as Trusted

Tool arguments are untrusted input.

Validate everything.

3. Returning Entire Backend Objects

Return only the fields necessary for the task.

4. Giving Every User Every Tool

Tool availability should reflect authorization.

5. Starting With High-Risk Write Operations

Begin read-only where possible.

Introduce mutations after your security and auditing model is mature.

6. Ignoring Observability

When an agent behaves unexpectedly, you need enough telemetry to reconstruct the interaction.

7. Putting Business Logic in MCP Handlers

Keep the MCP layer thin and move business operations into reusable services.

What MCP Changes and What It Doesn't

MCP solves an important integration problem:

How can AI applications interact with external
systems through a standardized interface?

It does not automatically solve:

Who should have access?

What data should the model see?

Which operations are safe?

Should an action require confirmation?

How should credentials be managed?

What should be logged?

How do we recover from failures?

Those remain engineering and security decisions.

That's especially important for internal tools because the systems being exposed may control customer data, production infrastructure, financial information, or operational workflows.

Final Takeaway

A good MCP server implementation isn't simply an adapter around an existing REST API.

It's a carefully designed boundary between AI applications and real systems.

Start small:

1. Identify a real internal workflow
2. Define narrow MCP tools
3. Validate every input
4. Return minimum necessary data
5. Start with read-only operations
6. Enforce authorization
7. Add timeouts and error handling
8. Audit important actions
9. Test failure and abuse scenarios
10. Expand only after the boundary is trustworthy

The code required to expose the first MCP tool can be surprisingly small.

The real engineering work is deciding what the AI should be allowed to do.

Get that boundary right, and MCP can turn disconnected internal APIs and services into a reusable tool layer for AI applications without requiring every new assistant or agent to reinvent those integrations.

Step-by-Step: Building an LLM Eval Harness From Scratch

Samcorp — Thu, 27 Aug 2026 08:48:58 GMT

Build a lightweight evaluation system that runs repeatable LLM tests, scores responses, tracks failures, and catches regressions before they reach production.

LLM applications are surprisingly easy to demo and surprisingly difficult to measure.

You change a system prompt and the answers look better.

Then you switch models and three edge cases quietly get worse.

You modify retrieval settings and hallucinations decrease—but response latency jumps.

Without structured evaluation, development quickly becomes:

Change something → try a few prompts → read the answers → decide whether it "feels better."

That works during the first prototype. It does not work when an LLM becomes part of a production system.

What we need instead is an eval harness: infrastructure that repeatedly executes test cases, captures outputs, grades them, and turns the results into something we can compare.

In this tutorial, we'll build an LLM eval harness from scratch using Python.

By the end, we'll have something like this:

Evaluation Dataset
       |
       v
+------------------+
|   Eval Runner    |
+------------------+
       |
       v
+------------------+
|  LLM Under Test  |
+------------------+
       |
       v
+------------------+
|     Graders      |
+------------------+
       |
       v
+------------------+
| Results + Logs   |
+------------------+
       |
       v
 Regression Report

No large evaluation framework is required to understand the fundamentals.

Let's build the pieces ourselves.

What Exactly Is an LLM Eval Harness?

An evaluation harness is the infrastructure responsible for running evaluations consistently.

At minimum, it needs to answer four questions:

What should we test?
How do we execute the model?
How do we determine whether the result is good?
How do we compare this run with previous runs?

That distinction matters.

An eval dataset contains test cases.

A grader evaluates an individual result.

An eval suite groups related tests.

The eval harness connects everything together: loading tests, executing requests, collecting responses, invoking graders, recording metadata, and aggregating results.

For teams building production LLM applications, evaluation should ideally be part of the broader development lifecycle rather than a final QA activity. This is particularly important when building generative AI applications where prompts, models, retrieval, tools, and application logic can all affect output quality.

Step 1: Define What "Good" Means

Don't start by writing the runner.

Start with the behavior you want to measure.

Suppose we're building a customer-support assistant.

We might care about:

Correctness
Instruction following
Groundedness
Required information
Formatting
Safety
Latency
Cost

A vague requirement such as:

The assistant should give good answers.

is almost impossible to evaluate.

Instead, convert it into observable requirements:

The assistant must answer using the supplied policy.
The assistant must not invent refund rules.
The assistant must mention the refund window.
The response must not exceed 150 words.

Now we have properties that can actually be tested.

This is the first major principle of eval design:

Evaluate specific behaviors, not a vague impression of quality.

Step 2: Create the Evaluation Dataset

Let's start with JSON.

Create:

evals.json

Then add a few cases:

[
  {
    "id": "refund_001",
    "category": "refund",
    "input": "Can I return an unopened product after 14 days?",
    "expected_contains": [
      "return",
      "14"
    ]
  },
  {
    "id": "password_001",
    "category": "account",
    "input": "How do I reset my password?",
    "expected_contains": [
      "reset",
      "password"
    ]
  },
  {
    "id": "shipping_001",
    "category": "shipping",
    "input": "How can I track my order?",
    "expected_contains": [
      "track",
      "order"
    ]
  }
]

This is intentionally simple.

A production dataset could contain considerably more information:

{
  "id": "refund_001",
  "category": "refund",
  "input": "...",
  "context": "...",
  "reference_answer": "...",
  "expected_contains": [],
  "forbidden_contains": [],
  "difficulty": "medium",
  "tags": ["policy", "refund"]
}

Where Should Eval Cases Come From?

Your strongest eval cases often come from actual product behavior:

production failures
user complaints
support tickets
manually discovered edge cases
expected product behaviors
adversarial inputs
previously fixed bugs

Don't create 1,000 artificial examples simply because 1,000 sounds more scientific.

A smaller set of meaningful tests is usually more useful at the beginning.

Step 3: Build a Model Adapter

We don't want evaluation logic tightly coupled to one model provider.

Instead of doing this everywhere:

response = some_provider_specific_call(...)

create a small abstraction:

class ModelAdapter:
    def generate(self, prompt: str) -> str:
        raise NotImplementedError

Then implement the provider:

class MyLLM(ModelAdapter):

    def __init__(self, client, model):
        self.client = client
        self.model = model

    def generate(self, prompt: str) -> str:

        response = self.client.responses.create(
            model=self.model,
            input=prompt
        )

        return response.output_text

Now the rest of our evaluation code doesn't need to know which provider we're using.

That makes experiments much easier:

Model A
Model B
Model C
New model version
Different temperature
Different system prompt

can all run through the same harness.

Step 4: Write the First Deterministic Grader

Before reaching for another LLM to grade responses, use deterministic checks whenever they can reliably measure the requirement.

Let's create a keyword grader:

def contains_grader(output, expected):

    output = output.lower()

    matches = [
        item.lower() in output
        for item in expected
    ]

    return {
        "score": sum(matches) / len(matches),
        "passed": all(matches)
    }

For example:

output = """
You can reset your password from the account settings page.
"""

expected = ["reset", "password"]

Result:

{
  "score": 1.0,
  "passed": true
}

Deterministic graders are excellent for requirements such as:

JSON validity
Required fields
Exact values
Regex patterns
Schema validation
Maximum length
Forbidden phrases
Tool outcomes
Database state
Unit tests

They're fast, inexpensive, repeatable, and easy to debug.

Step 5: Add Multiple Graders

A single metric rarely captures everything we care about.

Let's introduce another grader:

def length_grader(output, max_words=150):

    word_count = len(output.split())

    return {
        "word_count": word_count,
        "passed": word_count <= max_words
    }

And perhaps a forbidden-content grader:

def forbidden_phrase_grader(output, phrases):

    output_lower = output.lower()

    found = [
        phrase
        for phrase in phrases
        if phrase.lower() in output_lower
    ]

    return {
        "found": found,
        "passed": len(found) == 0
    }

Our evaluation can now measure multiple dimensions instead of compressing everything into "good" or "bad."

Step 6: Build the Eval Runner

Now we're ready for the core harness.

import json
import time

def load_evals(path):

    with open(path, "r") as file:
        return json.load(file)


def run_eval_case(model, case):

    start = time.perf_counter()

    output = model.generate(case["input"])

    latency = time.perf_counter() - start

    keyword_result = contains_grader(
        output,
        case.get("expected_contains", [])
    )

    length_result = length_grader(output)

    return {
        "id": case["id"],
        "category": case["category"],
        "input": case["input"],
        "output": output,
        "latency_seconds": latency,
        "graders": {
            "keywords": keyword_result,
            "length": length_result
        }
    }

Then execute the complete suite:

def run_suite(model, cases):

    results = []

    for case in cases:

        try:
            result = run_eval_case(model, case)
            results.append(result)

        except Exception as error:
            results.append({
                "id": case["id"],
                "error": str(error)
            })

    return results

Notice the try/except.

One failed API request should not destroy an entire evaluation run.

Step 7: Aggregate the Results

Individual cases are useful for debugging.

Aggregate results are useful for decisions.

def summarize(results):

    completed = [
        r for r in results
        if "error" not in r
    ]

    if not completed:
        return {
            "total": len(results),
            "completed": 0
        }

    keyword_scores = [
        r["graders"]["keywords"]["score"]
        for r in completed
    ]

    latencies = [
        r["latency_seconds"]
        for r in completed
    ]

    return {
        "total": len(results),
        "completed": len(completed),
        "keyword_score": (
            sum(keyword_scores) / len(keyword_scores)
        ),
        "average_latency": (
            sum(latencies) / len(latencies)
        )
    }

An evaluation run could now produce:

{
  "total": 50,
  "completed": 50,
  "keyword_score": 0.91,
  "average_latency": 1.84
}

That already gives us something much more useful than manually reading five responses.

Step 8: Save Every Run

Evaluation becomes much more powerful when results are historical.

from datetime import datetime

def save_results(results):

    timestamp = datetime.now().strftime(
        "%Y%m%d_%H%M%S"
    )

    path = f"results/eval_{timestamp}.json"

    with open(path, "w") as file:
        json.dump(results, file, indent=2)

    return path

Now we can compare:

Prompt v1 → 82%
Prompt v2 → 89%
Prompt v3 → 91%

Or:

Model A → 93% / 2.1 sec
Model B → 91% / 1.2 sec
Model C → 95% / 4.8 sec

Suddenly model selection becomes an engineering decision rather than a preference.

Step 9: Track Cost and Tokens

Quality isn't the only thing worth evaluating.

Production LLM systems also operate under constraints.

Extend the adapter so it returns metadata:

{
    "text": "...",
    "input_tokens": 430,
    "output_tokens": 120,
    "model": "model-name"
}

Then calculate:

cost = (
    input_tokens * input_rate
    + output_tokens * output_rate
)

Your report can now compare:

Version	Quality	Latency	Cost/Request
A	0.88	1.4s	$0.006
B	0.92	2.0s	$0.012
C	0.91	1.2s	$0.007

This prevents a common mistake: optimizing quality while ignoring the operational price of achieving it.

Step 10: Add LLM-as-a-Judge

Some properties cannot be captured reliably with string matching.

Consider:

Is the answer helpful?
Is it grounded in the supplied context?
Did it correctly explain the policy?
Did it hallucinate?
Does the answer satisfy the user's request?

These require semantic evaluation.

We can use another model as a grader.

def judge_response(judge_model, question, answer):

    prompt = f"""
You are evaluating an AI assistant.

Question:
{question}

Answer:
{answer}

Score the answer from 1 to 5.

Evaluate:
- correctness
- relevance
- completeness

Return JSON only:

{{
  "score": 1,
  "reason": ""
}}
"""

    return judge_model.generate(prompt)

For production use, make the rubric much more specific.

Instead of:

Is this a good answer?

prefer:

Score factual correctness from 1-5.

5 = completely correct
4 = correct with a minor omission
3 = partially correct
2 = major factual problems
1 = incorrect

A judge is itself an LLM, so its verdict isn't automatically ground truth.

Validate judge behavior against human-labeled examples before trusting it.

Step 11: Separate Capability Evals From Regression Evals

These serve different purposes.

Capability Evals

Capability evals answer:

How well can the system perform this behavior?

You intentionally include difficult examples.

A low initial score isn't necessarily bad—it gives the team room to improve.

Regression Evals

Regression evals answer:

Did something we changed break behavior that previously worked?

These tests should normally remain highly stable.

For example:

evals/
├── capability/
│   ├── complex_reasoning.json
│   └── difficult_support.json
│
└── regression/
    ├── refund_policy.json
    ├── formatting.json
    └── known_failures.json

Whenever you fix an important production failure, consider converting it into a regression case.

That's how your eval suite becomes more valuable over time.

Step 12: Add Category-Level Reporting

A global score can hide serious failures.

Imagine:

Overall score: 91%

Looks great.

But underneath:

General FAQ       99%
Shipping          97%
Account            95%
Refund             72%
Safety             61%

Now the picture is completely different.

Aggregate results by category:

from collections import defaultdict

def category_scores(results):

    categories = defaultdict(list)

    for result in results:

        if "error" in result:
            continue

        score = result["graders"]["keywords"]["score"]

        categories[result["category"]].append(score)

    return {
        category: sum(scores) / len(scores)
        for category, scores in categories.items()
    }

This tells us where the system is improving or deteriorating.

Step 13: Make Runs Reproducible

If two runs aren't comparable, the resulting numbers aren't very useful.

Record the configuration used for every run:

{
  "run_id": "2026-08-27-001",
  "model": "model-name",
  "temperature": 0,
  "prompt_version": "support-v7",
  "dataset_version": "3.2",
  "grader_version": "2.1",
  "commit": "abc123"
}

You should be able to answer:

"Why did our score fall from 92% to 86%?"

without guessing.

Possible causes might include:

Model changed
Prompt changed
Dataset changed
Retriever changed
Tool behavior changed
Grader changed
Harness changed

Version them.

Step 14: Keep Evaluation Environments Isolated

This becomes especially important when evaluating agents.

Imagine an agent that:

creates files,
modifies a database,
calls tools,
updates application state.

If the next evaluation starts with artifacts left by the previous run, the cases aren't independent anymore.

Each trial should therefore begin from a known state:

def setup_environment():
    reset_database()
    clear_temp_files()
    seed_test_data()


def run_isolated_case(case):

    setup_environment()

    try:
        return execute(case)

    finally:
        cleanup_environment()

For larger systems, this may involve:

containers
temporary databases
mock APIs
filesystem snapshots
sandbox environments

The closer your evaluation environment resembles the relevant parts of production, the more useful your results become.

Step 15: Don't Over-Grade the Agent's Path

Suppose an agent needs to find a customer's order.

A brittle eval might require:

1. call lookup_customer
2. call get_orders
3. call get_order_status
4. generate response

But what if the agent discovers an equally valid two-step solution?

It shouldn't fail simply because it solved the task differently.

When possible, grade the outcome:

assert order_status == "shipped"
assert response_contains_tracking_number

rather than requiring one exact reasoning trajectory.

For agents, the final environment state can often be more meaningful than the final text response.

Step 16: Turn Production Failures Into Tests

This may be the most valuable habit in the entire process.

Suppose a user discovers:

The assistant invents a refund policy
when an order is older than 90 days.

Don't just patch the prompt.

Create:

regression/refund_over_90_days.json

Then fix the system.

Now that failure can never silently return without showing up in your evaluation results.

Over time:

Production failure
       ↓
Root-cause analysis
       ↓
New eval case
       ↓
System fix
       ↓
Regression test
       ↓
Continuous evaluation

Your evaluation dataset becomes a record of what your system has learned not to break.

Teams developing larger LLM platforms can apply the same principle across model selection, RAG, prompting, integrations, validation, and monitoring. A broader LLM development workflow should therefore treat evaluation as an ongoing engineering loop rather than a one-time benchmark.

Step 17: Add the Harness to CI

Eventually, evals should run automatically.

A simple CI rule might be:

if regression_score < 0.95:
    raise SystemExit("Regression threshold failed")

Or compare against the baseline:

if new_score < baseline_score - 0.02:
    raise SystemExit("Quality regression detected")

A deployment workflow might become:

Developer changes prompt/model/RAG
            |
            v
       Pull Request
            |
            v
       Unit Tests
            |
            v
       LLM Evals
            |
       +----+----+
       |         |
      Pass      Fail
       |         |
       v         v
     Merge     Review

Don't necessarily block deployments on every experimental capability metric.

Regression suites are usually better suited for hard CI thresholds.

A Better Project Structure

Once the harness grows, keep its components separate.

llm-evals/
│
├── datasets/
│   ├── capability.json
│   └── regression.json
│
├── graders/
│   ├── keyword.py
│   ├── schema.py
│   ├── judge.py
│   └── safety.py
│
├── models/
│   └── adapter.py
│
├── runners/
│   └── runner.py
│
├── reports/
│   └── summary.py
│
├── results/
│
├── config.yaml
│
└── main.py

This makes it easier to modify one part without quietly changing the others.

What Should You Log?

At minimum, save:

Run ID
Test case ID
Dataset version
Model
Model configuration
Prompt version
Input
Context
Output
Grader results
Latency
Token usage
Cost
Errors
Timestamp

For agentic applications, you may additionally need:

Tool calls
Tool results
Number of turns
Retries
Final environment state
Resource budget

Logs aren't merely debugging information.

They become raw material for future evals.

Common Mistakes When Building an Eval Harness

1. Testing Only Happy Paths

If every example is easy, your score tells you very little.

Include boundary cases and known failures.

2. Using Only One Aggregate Score

A 94% average can hide a critical category running at 60%.

Break results down by behavior.

3. Using LLM Judges for Everything

Use deterministic evaluation when the requirement itself is deterministic.

Save model-based grading for semantic judgments.

4. Changing the Harness During Model Comparisons

If Model A gets different tools, prompts, retries, or budgets than Model B, you're no longer performing a clean comparison.

5. Trusting the Grader Without Testing It

A broken grader produces confident-looking but meaningless numbers.

Validate graders too.

6. Ignoring Variability

LLM outputs are probabilistic.

For important tests, consider multiple trials rather than assuming one response represents expected behavior.

7. Building Evals Too Late

If evaluation begins only after launch, you're forced to reconstruct expected behavior from an already complex system.

Start small and start early.

From Tiny Script to Production Eval Platform

Our first version was essentially:

JSON
  ↓
Model
  ↓
Grader
  ↓
Score

A mature system might look more like:

Versioned datasets
       ↓
Parallel execution
       ↓
Model / Agent
       ↓
Deterministic graders
       +
LLM judges
       +
Environment checks
       ↓
Results database
       ↓
Dashboards
       ↓
Baseline comparison
       ↓
CI/CD quality gate

The architecture becomes more sophisticated, but the core idea doesn't change:

Define expected behavior, execute consistently, measure the result, preserve the evidence, and compare changes.

Final Takeaway

If you want to build an LLM eval harness, don't begin by searching for the perfect evaluation framework.

Begin with five pieces:

1. A small dataset of meaningful tasks
2. A repeatable model runner
3. Simple, trustworthy graders
4. Detailed result logging
5. Baselines for comparison

Then improve the system as your application grows.

Add semantic judges when deterministic checks aren't enough.

Add isolated environments when agents modify state.

Add multiple trials when variance matters.

Add cost and latency when production economics matter.

Add regression gates when your team starts shipping frequently.

Most importantly, turn real failures into permanent tests.

The goal isn't to create an impressive evaluation dashboard.

The goal is to know whether your LLM application is actually getting better.

Automating Odoo Deployments With Docker and CI

Samcorp — Wed, 19 Aug 2026 12:25:38 GMT

Deploying Odoo manually can work for small projects, but as custom modules and environments grow, repeatable deployments become far more valuable.

With an Odoo Docker deployment, you can package Odoo and its dependencies into a consistent container, while CI/CD can automatically test, build, publish, and deploy each approved change.

A practical workflow looks like this:

Developer Push
      ↓
CI Pipeline
      ↓
Validate & Test
      ↓
Build Docker Image
      ↓
Push to Registry
      ↓
Deploy to Server
      ↓
Health Check

Odoo maintains an official Docker image repository, including current Docker definitions for Odoo 17, 18, and 19. The Odoo 19 image also provides dedicated paths for persistent application data and custom addons.

Why Use Docker for Odoo Deployment?

Without containers, two Odoo servers can slowly become different because of:

Python packages
System dependencies
Odoo configuration
Custom addon versions
OS-level changes

Docker gives the deployment a more consistent runtime.

Instead of repeatedly configuring the server manually, you define the environment as code:

Dockerfile
+
Compose configuration
+
Odoo configuration
+
Custom addons

The same image can then move through testing, staging, and production.

Step 1: Organize the Odoo Project

A simple project structure could be:

odoo-project/
├── custom-addons/
├── config/
│   └── odoo.conf
├── Dockerfile
├── compose.yaml
├── compose.production.yaml
└── .github/
    └── workflows/
        └── deploy.yml

Keep custom modules in version control so every deployment is connected to an exact code revision.

Step 2: Create the Odoo Dockerfile

For Odoo 19, you can build on the Odoo image and add your own modules.

FROM odoo:19.0

COPY ./custom-addons /mnt/extra-addons
COPY ./config/odoo.conf /etc/odoo/odoo.conf

The official Odoo 19 Docker definition uses /mnt/extra-addons for user addons and /var/lib/odoo for persistent Odoo data.

For real projects, keep the image focused.

Avoid installing unnecessary packages just because they may be useful later.

A structured Odoo development and deployment workflow can combine Docker-based environments, staging, automated CI/CD pipelines, and controlled production releases.

Step 3: Add PostgreSQL With Docker Compose

Odoo requires PostgreSQL, so Compose can define both services.

services:

  db:
    image: postgres:16
    restart: unless-stopped

    environment:
      POSTGRES_DB: postgres
      POSTGRES_USER: odoo
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}

    volumes:
      - postgres_data:/var/lib/postgresql/data

    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U odoo"]
      interval: 10s
      timeout: 5s
      retries: 5

  odoo:
    build: .
    restart: unless-stopped

    depends_on:
      db:
        condition: service_healthy

    ports:
      - "8069:8069"

    environment:
      HOST: db
      USER: odoo
      PASSWORD: ${POSTGRES_PASSWORD}

    volumes:
      - odoo_data:/var/lib/odoo

volumes:
  postgres_data:
  odoo_data:

The health check is useful because starting PostgreSQL does not automatically mean it is ready to accept connections. Docker Compose supports condition: service_healthy so a dependent application can wait for a dependency's health check to succeed.

Step 4: Keep Important Data Persistent

Containers should be replaceable.

Your business data should not disappear when a container is recreated.

Persist at least:

PostgreSQL data
Odoo filestore

The official Odoo image exposes /var/lib/odoo specifically for persistent Odoo data, including the filestore.

The deployment model should therefore be:

Disposable Container
        +
Persistent Database
        +
Persistent Filestore

This makes rebuilding the Odoo container much safer.

Step 5: Separate Development and Production Configuration

Production usually needs different settings from local development.

Docker recommends using an additional Compose configuration for production-specific changes.

For example:

compose.yaml
compose.production.yaml

You might use the production file to:

Remove unnecessary exposed ports
Configure restart policies
Use production images
Connect to a reverse proxy
Apply production resources or volumes

Then deploy with:

docker compose \
  -f compose.yaml \
  -f compose.production.yaml \
  up -d

This keeps environment-specific changes clear.

Step 6: Keep Secrets Out of the Image

Never write production credentials directly into your Dockerfile:

ENV POSTGRES_PASSWORD=super-secret-password

Avoid committing them to Git as well.

Docker recommends using dedicated secrets mechanisms for sensitive values rather than ordinary environment variables where possible.

Sensitive values may include:

PostgreSQL password
Odoo master password
Registry credentials
SSH keys
API tokens
SMTP passwords

Your CI system should also keep deployment credentials in its protected secret store.

Step 7: Add CI Before Automatic Deployment

The important idea behind CI is simple:

Don't deploy code that has not passed validation.

A useful pipeline might be:

Git Push
   ↓
Checkout
   ↓
Build
   ↓
Run Tests
   ↓
Build Production Image
   ↓
Push Image
   ↓
Deploy

Docker provides official GitHub Actions specifically for building and publishing container images, and GitHub also documents workflows for publishing images to Docker registries.

Step 8: Create a GitHub Actions Workflow

Here is a simplified example.

name: Odoo CI

on:
  push:
    branches:
      - production

jobs:

  build:
    runs-on: ubuntu-latest

    steps:

      - name: Checkout repository
        uses: actions/checkout@v6

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build Odoo image
        uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: my-odoo:test

At this point, the workflow proves that the Docker image can be built.

That alone already catches problems such as:

Invalid Dockerfile instructions
Missing copied files
Broken dependencies
Build-time failures

Step 9: Test Before Publishing

A better pipeline should validate the image before pushing it to your registry.

Docker provides a documented test-before-push workflow pattern specifically for this reason.

For Odoo, testing might include:

odoo \
  -d test_db \
  -i custom_module \
  --test-enable \
  --stop-after-init

Depending on your project, you can test:

Module installation
Python tests
Access rights
XML loading
Business workflows
Upgrade scripts

Your pipeline becomes:

Build
   ↓
Test
   ↓
Pass?
 ┌─┴─┐
No  Yes
│     │
Stop  Push Image

This prevents a large class of deployment problems from reaching production.

Step 10: Push the Approved Image

After validation passes, authenticate to your container registry.

- name: Login to registry
  uses: docker/login-action@v3
  with:
    username: ${{ secrets.REGISTRY_USERNAME }}
    password: ${{ secrets.REGISTRY_PASSWORD }}

Then publish:

- name: Build and push
  uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: |
      company/odoo:${{ github.sha }}
      company/odoo:production

Using the Git commit SHA as an image tag is helpful:

company/odoo:a5b3c74...

because it gives you a clear connection between:

Running container
      ↓
Docker image
      ↓
Git commit

That makes debugging and rollback easier.

Step 11: Deploy the New Image

Once the registry contains the approved image, production can pull it.

For example:

docker compose pull odoo
docker compose up -d --no-deps odoo

Docker's production Compose guidance supports recreating an updated service without unnecessarily recreating its dependencies.

Your PostgreSQL container and persistent volumes remain separate from the replaceable Odoo application container.

Step 12: Add a Deployment Health Check

Automation should not end with:

Container started

You also want to know:

Is Odoo actually responding?

A simple deployment check might be:

curl --fail http://localhost:8069/web/login

Conceptually:

Deploy
   ↓
Wait for Odoo
   ↓
Health Check
   ↓
Success → Finish
Failure → Investigate/Roll Back

This gives the pipeline a clear success condition.

Step 13: Put a Reverse Proxy in Front of Odoo

For an internet-facing production deployment, avoid treating Odoo's application port as your complete production web stack.

Odoo's production deployment documentation covers deployment behind a proxy and production configuration such as proxy_mode.

A common architecture is:

Internet
   ↓
HTTPS
   ↓
Nginx / Reverse Proxy
   ↓
Odoo Container
   ↓
PostgreSQL

The reverse proxy can handle:

TLS
Domain routing
Request forwarding
Security headers

Your PostgreSQL service generally remains internal to the Docker network rather than being exposed publicly.

Step 14: Make Rollback Part of CI/CD

Automation is valuable only when recovery is also predictable.

Suppose the new version is:

company/odoo:8fd21ab

and the previous stable version was:

company/odoo:42ac718

A rollback can switch the application image back to:

company/odoo:42ac718

and recreate the Odoo service.

However, remember an important Odoo-specific point:

Container rollback and database rollback are not always the same thing.

If a deployment upgraded modules or changed database structures, simply returning to an older Docker image may not be enough.

For database-changing releases, take a verified backup before deployment.

For upgrades or database-changing releases, Odoo migration and deployment automation can combine reproducible Docker environments, automated validation, backups, and rollback planning before production cutover.

A Practical Odoo Docker Deployment Pipeline

The final process could look like this:

Developer
   │
   ▼
Git Push
   │
   ▼
CI Pipeline
   │
   ├── Build Docker Image
   │
   ├── Start Test Database
   │
   ├── Install Modules
   │
   └── Run Tests
   │
   ▼
Push Versioned Image
   │
   ▼
Production Server
   │
   ├── Backup
   ├── Pull Image
   ├── Deploy
   └── Health Check
   │
   ▼
Odoo Live

This creates a much clearer deployment history than manually copying addons directly onto a production server.

Common Odoo Docker Deployment Mistakes

Using `latest` for everything

Use predictable versioning so you know exactly which image is running.

Keeping PostgreSQL inside the Odoo container

Keep database and application responsibilities separate.

Forgetting persistent volumes

Containers are disposable. Business data is not.

Deploying before testing

Build and validate before publishing the production image.

Storing credentials in Git

Use your CI/CD secret store and appropriate runtime secret management.

Updating modules without a backup

Database-changing deployments deserve a verified rollback plan.

Rebuilding directly on production

Build once in CI and promote the tested artifact whenever possible.

Final Thoughts

A reliable Odoo Docker deployment becomes much easier when deployment is treated as a repeatable pipeline rather than a sequence of manual server commands.

A solid starting point is:

Code → Test → Build → Push → Backup → Deploy → Verify

Docker gives you a consistent Odoo runtime, while CI/CD gives every change the same validation and deployment process. Docker's official CI tooling supports automated image building and publishing, and its production Compose guidance supports controlled service updates.

Start with a simple Odoo + PostgreSQL container setup.

Then automate one stage at a time.

Once your pipeline consistently builds, tests, versions, deploys, and verifies the same artifact, Odoo releases become easier to repeat, audit, and recover.

Building Your First Odoo Connector: Shopify Order Sync

Samcorp — Tue, 18 Aug 2026 11:34:45 GMT

A well-designed Odoo Shopify integration can automatically bring Shopify orders into Odoo and reduce repetitive manual work.

For a first connector, the best approach is to keep the workflow simple:

Shopify Order
   ↓
orders/create Webhook
   ↓
Verify HMAC
   ↓
Check Duplicate
   ↓
Map Customer & Product
   ↓
Create Odoo Sales Order
   ↓
Store Shopify Order ID

This gives you a reliable foundation that can later be extended to inventory, fulfillment, refunds, and customer synchronization.

Step 1: Create the Odoo Connector Module

Start with a small custom Odoo module.

shopify_odoo_connector/
├── __init__.py
├── __manifest__.py
├── controllers/
├── models/
├── security/
└── views/

Your module can depend on the main applications required for order creation:

{
    "name": "Shopify Odoo Connector",
    "version": "1.0.0",
    "depends": [
        "sale_management",
        "stock",
        "contacts",
    ],
    "installable": True,
}

Keeping the first version focused makes development and testing much easier.

Step 2: Configure the Shopify Store

Create a model for the Shopify instance.

from odoo import fields, models


class ShopifyInstance(models.Model):
    _name = "shopify.instance"
    _description = "Shopify Instance"

    name = fields.Char(required=True)
    shop_domain = fields.Char(required=True)
    active = fields.Boolean(default=True)

Store Shopify credentials securely rather than placing access tokens directly inside Python code.

Step 3: Receive Shopify Orders With a Webhook

When a new Shopify order is created, Shopify can send an orders/create webhook to Odoo.

Example endpoint:

POST /shopify/webhook/orders/create

A basic Odoo controller may look like:

from odoo import http
from odoo.http import request


class ShopifyWebhookController(http.Controller):

    @http.route(
        "/shopify/webhook/orders/create",
        type="http",
        auth="public",
        methods=["POST"],
        csrf=False,
    )
    def shopify_order_create(self, **kwargs):

        raw_body = request.httprequest.data

        # Verify and process the webhook

        return request.make_response("OK", status=200)

Using webhooks allows orders to move into Odoo soon after they are created.

For a production-ready reference, this Shopify to Odoo Connector demonstrates webhook-based synchronization for orders, customers, products, and inventory.

Step 4: Verify the Shopify Webhook

Before processing an order, verify Shopify's HMAC signature.

Shopify sends the signature in:

X-Shopify-Hmac-SHA256

A simplified verification function is:

import base64
import hashlib
import hmac


def verify_webhook(raw_body, received_hmac, secret):

    digest = hmac.new(
        secret.encode(),
        raw_body,
        hashlib.sha256,
    ).digest()

    calculated = base64.b64encode(digest).decode()

    return hmac.compare_digest(
        calculated,
        received_hmac,
    )

This helps ensure that your Odoo endpoint processes genuine Shopify requests.

Step 5: Prevent Duplicate Orders

Webhook events can occasionally be delivered more than once.

Use Shopify's webhook ID and Shopify order ID to make the integration idempotent.

Before creating an Odoo order:

existing_order = env["sale.order"].search([
    ("shopify_order_id", "=", str(payload["id"]))
], limit=1)

If the order already exists, simply skip the duplicate request.

This ensures:

1 Shopify Order
      ↓
1 Odoo Order

Step 6: Match Shopify Products With Odoo

A simple starting point is matching the Shopify SKU with the Odoo internal reference.

product = env["product.product"].search([
    ("default_code", "=", shopify_sku)
], limit=1)

For example:

Shopify SKU: TSHIRT-BLACK-M
        ↓
Odoo Internal Reference: TSHIRT-BLACK-M

If a product cannot be matched, log the issue clearly instead of silently creating an incorrect product.

Step 7: Find or Create the Customer

Search for the Shopify customer in Odoo using a reliable identifier such as email or a stored Shopify customer ID.

partner = env["res.partner"].search([
    ("email", "=", customer_email)
], limit=1)

If no customer exists:

partner = env["res.partner"].create({
    "name": customer_name,
    "email": customer_email,
    "phone": customer_phone,
})

Storing Shopify's customer ID in Odoo can make future synchronization even more reliable.

Step 8: Create the Odoo Sales Order

Once products and customers are mapped, build the order lines.

order_lines.append((
    0,
    0,
    {
        "product_id": product.id,
        "product_uom_qty": quantity,
        "price_unit": price,
    }
))

Then create the sales order:

sale_order = env["sale.order"].create({
    "partner_id": partner.id,
    "client_order_ref": shopify_order_name,
    "order_line": order_lines,
})

For an initial connector, keeping imported orders as quotations can be useful while you validate payment, tax, inventory, and shipping rules.

Step 9: Store Shopify References

Save the external Shopify order ID in Odoo.

shopify_order_id = fields.Char(
    index=True,
    copy=False,
)

This helps with:

Duplicate prevention
Order updates
Cancellations
Refunds
Fulfillment synchronization

Stable external IDs make future connector features much easier to manage.

Step 10: Add Background Processing

For a small prototype, orders can be processed directly.

For production, a stronger architecture is:

Webhook Received
      ↓
Verify HMAC
      ↓
Store Event
      ↓
Return Success
      ↓
Process in Background
      ↓
Create Odoo Records

This keeps webhook responses fast and makes order processing easier to retry if an issue occurs.

Step 11: Add a Reconciliation Job

Webhooks are excellent for real-time events, but a scheduled reconciliation job adds another layer of reliability.

For example:

Webhook Sync
    +
Hourly Reconciliation

The reconciliation process can compare recently updated Shopify orders with Odoo and recover anything that was missed during temporary downtime.

You can also follow this detailed guide to configure a Shopify Odoo connector, including API authentication, store registration, SKU mapping, order synchronization, and scheduler setup.

Testing Your Odoo Shopify Integration

Before going live, test a few important scenarios.

Standard Order

Known customer + known product.

Expected: One Odoo quotation is created.

Multiple Products

Several Shopify order lines.

Expected: All products appear correctly in one Odoo order.

Unknown SKU

Expected: The connector records a clear mapping issue.

Duplicate Webhook

Send the same event twice.

Expected: Only one Odoo order exists.

Invalid HMAC

Expected: The request is rejected and no record is created.

These tests help confirm that the connector behaves safely before real orders are synchronized.

Common Integration Mistakes to Avoid

Keep these points in mind:

Store API credentials securely
Verify every Shopify webhook
Prevent duplicate order creation
Use stable product identifiers
Log mapping issues clearly
Test taxes, shipping, and payments before automatic confirmation
Keep a reconciliation process for recovery

Each of these practices makes the integration easier to maintain as it grows.

What Can You Add Next?

Once Shopify order sync is stable, you can expand the connector gradually:

Phase 1: Orders
Phase 2: Customers
Phase 3: Products & Variants
Phase 4: Inventory
Phase 5: Order Updates
Phase 6: Fulfillment
Phase 7: Refunds
Phase 8: Monitoring & Reconciliation

A phased approach keeps the Odoo Shopify integration easier to test, debug, and improve.

Final Thoughts

Building your first Shopify connector becomes much simpler when you start with one reliable workflow.

Focus first on:

Verify → Deduplicate → Map → Import → Log

Once Shopify orders are consistently reaching Odoo, you can confidently extend the connector to customers, inventory, fulfillment, and refunds.

A small, dependable integration is a much stronger starting point than trying to synchronize everything at once.

Odoo 16 to 18 Upgrade: Step-by-Step Guide for Custom Modules

Samcorp — Tue, 18 Aug 2026 09:12:04 GMT

An Odoo 16 to 18 upgrade is more than a database migration when custom modules are involved. Your database may upgrade successfully, but custom Python code, XML views, JavaScript, integrations, and historical data can still break.

Odoo officially supports upgrading older databases to newer versions, including a 16.0 to 18.0 migration path. Custom code, however, still needs to be reviewed, updated, and tested separately.

Step 1: Audit Your Odoo 16 Setup

Before upgrading, make a list of everything customized in your current system:

Custom modules
Third-party apps
Odoo Studio changes
External integrations
Scheduled actions
Custom reports
Security rules
Payment or shipping connectors

Also remove customizations that are no longer needed. Odoo 18 may already provide some functionality that previously required custom development.

Step 2: Take a Full Backup

Create a complete backup before making any changes.

Back up:

PostgreSQL database
Odoo filestore
Custom addons
Configuration files
Environment variables
Integration settings

A typical PostgreSQL backup command is:

pg_dump -Fc production_db > odoo16_backup.dump

Always keep the matching filestore with the database backup.

Step 3: Create a Separate Odoo 18 Environment

Do not upgrade directly on your live Odoo 16 server.

Create a clean Odoo 18 staging environment and configure the required addon paths.

For example:

addons_path = /opt/odoo18/odoo/addons,/opt/odoo18/enterprise,/opt/odoo18/custom-addons

First confirm that Odoo 18 starts correctly before introducing migrated data.

Step 4: Make Custom Modules Odoo 18 Compatible

Test each custom module on a fresh Odoo 18 database.

./odoo-bin \
  -d odoo18_test \
  -i custom_module \
  --stop-after-init

Fix compatibility issues such as:

Removed or renamed fields
Changed model methods
Broken Python imports
Invalid XML views
XPath failures
Asset changes
JavaScript or OWL errors
Missing dependencies

If your environment contains heavily modified modules, reviewing them through an Odoo customization services workflow can help identify outdated dependencies, unsafe core modifications, view changes, and upgrade-sensitive business logic before migration.

Also update the module manifest where necessary.

{
    "name": "Custom Sales Extension",
    "version": "18.0.1.0.0",
    "depends": ["sale_management", "stock"],
    "installable": True,
}

A module should install cleanly on Odoo 18 before you test it with migrated production data.

Step 5: Review XML and Frontend Customizations

Custom views often fail because the parent Odoo view has changed.

For example:

If the Odoo 18 parent view is different, this XPath may stop working.

Review:

Form and list views
Search views
Kanban views
QWeb templates
Reports
OWL components
JavaScript patches
POS customizations

Use browser developer tools to identify frontend errors.

Step 6: Upgrade a Test Database

Once your custom modules are mostly compatible, create an upgraded test database using Odoo's supported upgrade process.

For complex environments involving custom modules, integrations, and production data, a structured Odoo migration services approach can help organize staging, validation, reconciliation, and rollback planning before the final cutover.

Do not move directly to production.

Use the test database to identify issues with:

Existing records
Custom fields
Accounting data
Inventory
Attachments
Workflows
Integrations

Make sure the correct filestore is also restored.

Step 7: Write Migration Scripts for Custom Data

Code changes do not automatically transform historical records.

Suppose Odoo 16 stores:

state = legacy

but the Odoo 18 module expects:

state = active

You may need an upgrade script:

def migrate(cr, version):
    cr.execute("""
        UPDATE custom_record
        SET state = 'active'
        WHERE state = 'legacy'
    """)

Migration scripts are commonly required when:

Fields are renamed
Models change
Selection values change
Data moves between models
Relationships change
Old fields are removed

Odoo also supports pre, post, and end migration phases for handling these transformations.

Step 8: Test Complete Business Workflows

A successful module installation does not mean the business process still works.

Test complete workflows such as:

Sales

Quotation → Sales Order → Delivery → Invoice → Payment

Purchase

RFQ → Purchase Order → Receipt → Vendor Bill

Inventory

Check receipts, deliveries, transfers, lots, serial numbers, and valuation.

Accounting

Validate taxes, journal entries, reconciliation, credit notes, payments, and reports.

Also test every custom workflow created specifically for your organization.

Step 9: Test External Integrations

Verify all connected services, including:

Payment gateways
Shipping providers
Marketplaces
REST APIs
Webhooks
SSO
BI tools
Email services

Check authentication, payload formats, scheduled synchronization, and error handling.

Step 10: Rehearse the Migration

Run the complete upgrade again using a fresh Odoo 16 backup.

Record how long each stage takes:

Backup
Database upgrade
Restore
Custom module update
Migration scripts
Testing
Final validation

This helps you plan a realistic production maintenance window.

Step 11: Prepare a Rollback Plan

Before production migration, make sure you can return to Odoo 16 if necessary.

Your rollback plan should include:

Final Odoo 16 database backup
Matching filestore backup
Original Odoo 16 server kept intact
Reversible proxy or DNS configuration
Clear rollback decision criteria

Step 12: Perform the Production Upgrade

Once staging is approved, follow a controlled sequence:

1. Stop user access
2. Take final backup
3. Upgrade the database
4. Deploy Odoo 18-compatible custom modules
5. Restore the filestore
6. Run migration scripts
7. Upgrade modules
8. Run smoke tests
9. Verify integrations
10. Reopen the system

Deploy only the code that has already been tested in staging.

Common Odoo 16 to 18 Upgrade Mistakes

Avoid these common problems:

Treating the upgrade as database-only
Testing directly in production
Migrating unnecessary custom modules
Ignoring the filestore
Skipping custom data migration
Checking only whether modules install
Forgetting integrations
Testing individual screens instead of complete workflows

Can You Upgrade Directly From Odoo 16 to Odoo 18?

Yes. Odoo supports upgrading a database from Odoo 16 to Odoo 18.

The important part is understanding that the database upgrade and custom-code migration are different tasks.

A safer workflow is:

Odoo 16
   ↓
Database Upgrade
   ↓
Odoo 18 Database
   +
Compatible Custom Modules
   +
Data Migration Scripts
   +
Business Testing

Final Thoughts

A reliable Odoo 16 to 18 upgrade should follow a controlled process:

Audit → Backup → Update Custom Modules → Upgrade Test Database → Migrate Data → Test → Rehearse → Go Live

The biggest mistake is assuming that a database that successfully starts on Odoo 18 is fully migrated.

Custom modules, integrations, accounting workflows, inventory processes, and existing data should all be tested before production cutover.

How to Choose the Best Odoo Customization Company in India: A Complete Guide

Samcorp — Tue, 01 Jul 2025 11:58:05 GMT

What is Odoo Development?

Odoo development refers to building, modifying, or extending the capabilities of the Odoo ERP platform. As an open-source and modular system, Odoo provides applications for finance, sales, inventory, HR, project management, and more. Through Odoo customization, businesses can tailor these modules to match unique workflows, integrate with third-party tools, or even develop completely new features. Professional Odoo development services include module customization, API integrations, migration, performance tuning, and deployment support to ensure that the ERP ecosystem functions smoothly and meets your precise operational requirements.

Why Do Businesses Need Odoo Development Companies?

While Odoo’s modular architecture is powerful, implementing and adapting it to complex, real-world business processes often requires specialized skills. This is where an experienced Odoo customization company becomes crucial. Such companies combine functional consultants with technical developers to:

Understand and map your business workflows
Perform efficient module configuration
Deliver Odoo customization services tailored to your business rules
Ensure data security and migration integrity
Provide long-term maintenance and support

Without expert guidance, implementing Odoo can result in poor configurations, data loss, or compatibility issues. Partnering with a professional team guarantees a robust, scalable, and future-ready ERP implementation.

What are the Types of Odoo Development Services Available?

Top Odoo development companies typically offer a broad range of services, including:

Odoo customization (tailoring standard modules to fit your workflows)
Custom module development (building new features from scratch)
Odoo migration services (upgrading to newer Odoo versions with secure data migration)
Third-party API integrations (connecting Odoo with external apps like payment gateways, CRMs, or shipping providers)
Performance optimization (improving load times and data processing speed)
Odoo training (enabling your staff to use the platform effectively)
Odoo support and maintenance (continuous updates, bug fixes, and upgrades)

These Odoo customization services empower businesses to extend Odoo’s default capabilities and align the system precisely with their operations.

Basic Factors When Choosing an Odoo Development Company

Choosing the best Odoo customization company in India requires assessing several critical factors:

Technical expertise: Verify their experience with Odoo’s frameworks (Python, PostgreSQL, XML) and modern development practices.
Functional knowledge: Ensure they have consultants who understand business domains like manufacturing, trading, e-commerce, or accounting.
Portfolio: Review their previous Odoo customization projects, complexity of integrations, and real-world results.
Certifications & partnerships: Prefer certified Odoo partners or official contributors to the Odoo community.
Communication & project management: Evaluate their ability to deliver milestones transparently and manage change requests effectively.
Post-deployment support: Confirm their service-level agreements for long-term maintenance, bug fixing, and updates.

Evaluating the Company’s Portfolio and Case Studies

A reputable Odoo customization company should have a clear, verifiable portfolio that demonstrates:

End-to-end Odoo implementations
Complex module customizations
Cross-industry experience
Successful Odoo migration projects
Scalability and performance improvements

Reviewing detailed case studies is an essential step. Ask for success stories relevant to your domain, measurable outcomes achieved, and references from their past clients. This due diligence will help you assess whether their Odoo customization service truly meets your needs.

Conclusion

Odoo is a powerful ERP platform, but to unlock its full potential, a skilled partner is essential. The best Odoo customization companies in India combine technical excellence, industry knowledge, and a commitment to ongoing support. By carefully evaluating their skills, portfolio, and support structures, you can select a reliable Odoo customization service provider that aligns with your business goals, ensuring a secure, scalable, and efficient ERP solution for years to come.