The Real Cost of Maintaining an AI System: The 18 Months After Launch

Once software has been launched, it usually becomes more predictable. When the main phase of development is finished, the system enters the maintenance stage and the team turns their focus to the next item on the roadmap.

It is not the case that AI systems always adhere to that pattern.

An AI system is affected not just by its code but also by its data, user behaviour, the quality of retrieval, model performance, and the models or services that it relies on; whenever any of these things change the system can also change.

It means that maintaining an AI system becomes the ongoing concern of engineering rather than being covered by a standard support contract.

The first 18 months following go-live are especially important since it is in this period that teams identify the difference between controlled testing and real-world use, set up production monitoring, optimize their operating costs, and get ready for changes in the underlying AI models.

That is how the lifecycle can appear.

Months 0 to 3: Closing the Gap Between Testing and Reality

Actual users will carry out things that your evaluation set did not anticipate.

Normally, they aren’t aiming at breaking the system but are merely making use of it in ways that were hard to reproduce during development.

Questions could be formulated in a way that is different from the examples used when the system was tested. The documents might include layouts or fields which the extraction process has only infrequently come across. It is possible that one department could make greater use of the system than had been anticipated and thereby introduce a workflow that was not originally included in the design.

The main activity in this stage consists of improving the system’s performance by converting the real-world cases.

The teams have to gather some production examples, categorise the different types of failure, extend the evaluation set, and then decide which of the problems arises from the model, from retrieval, from prompting, from data quality, from business logic, or from the user workflow.

The work is not very noticeable, though it is also one of the most important phases when it comes to building user trust.

A system which performs well in real-world edge cases is more likely to be adopted, while one that continually generates results that are plausible but unreliable tends to be workarounds rather than something that employees actually work with.

There isn’t a general maintenance percentage for this period, but it’s reasonable to expect a considerable amount of engineering work right after the launch rather than thinking that the project is complete when go-live takes place.

Months 3 to 6: Detecting the First Signs of Drift

Performance may begin to change even if no one has altered the application code.

The reason lies in the fact that the production data has changed.

The system might have to deal with documents that it has not come across before if a new product line is introduced. A change to a process could affect the way a form is filled out. The supplier might amend its invoice template. And users might also come up with new methods of interacting with the system after they have worked out what it is capable of.

The changes can cause the distribution of the inputs that the system gets to be altered.

It is usually called data drift or input drift. This is distinct from a situation in which a model suddenly becomes less capable. The environment surrounding the model has changed and the system is therefore no longer operating under the conditions that were represented in the original evaluation data.

There can also be concept drift, where the relationship between an input and the expected output changes. For example, a business rule or classification criterion may change even though the underlying document format stays the same.

The only sure method of identifying these changes is to keep an eye on the quality of production.

Monitoring should go beyond uptime, latency, and API errors. Depending on the use case, teams may track extraction accuracy, retrieval relevance, classification precision and recall, human override rates, escalation frequency, or other task-specific metrics.

A scheduled evaluation run is equally important. Running a representative evaluation set at regular intervals makes it possible to compare performance over time rather than judging each release in isolation.

Months 6 to 12: Where Cost Optimization Starts to Matter

At this stage, the way the language is used is generally more clear.

You know which workflows bring about the most traffic, which requests are fairly simple, which involve more context, and in which cases human review is still needed.

This provides an opportunity to tailor the architecture to be suitable for use in actual production.

A system may initially be designed around correctness and reliability rather than cost. Once real traffic data is available, teams can start looking at the economics of each part of the pipeline.

Common optimization strategies include:

• Routing simple requests to smaller or less expensive models.
• Caching repeated or predictable requests where appropriate.
• Reducing unnecessary context sent to the model.
• Improving retrieval so fewer irrelevant documents reach the generation stage.
• Moving deterministic tasks from an LLM to conventional software logic where possible.
• Using structured outputs to reduce downstream processing.
• Fine-tuning or distilling a smaller model when the task is narrow, high-volume, and sufficiently stable.

The amount of money that can be saved might be considerable, but there is no single percentage that applies to all cases of AI deployment. The result will vary according to the original architecture, the model pricing, the distribution of the traffic, the context size, and the extent to which the workload can be safely directed to cheaper processing.

That is also the reason why cost optimization cannot be separated from evaluation.

A model that is less expensive is only an improvement if the decrease in cost does not lead to an unacceptable rise in errors. It is the evaluation set that gives us the evidence required to make that trade-off deliberately.

Providers that plan for optimization as part of the delivery lifecycle, such as tailored AI software development – TechTIQ Inc., can address these issues before they become a separate operational problem. When reviewing a proposal, it is worth asking not only how the system will be launched, but also how model usage and infrastructure costs will be reviewed once production traffic is known.

Months 9 to 15: Preparing for Model Changes

The model behind an AI application does not have to be a permanent part of it.

AI companies might withdraw old model versions, launch new ones, alter their pricing, or release updated models which provide a different combination of capability, latency, and cost.

A migration of a model is therefore not simply a matter of altering an API parameter.

A replacement model could interpret prompts in a different way, generate output structures that are different, retrieve information in a different manner when used with a specific workflow, or cause a change in the application’s latency and cost profile.

The migration process must involve a new assessment based on representative production cases, and the teams might have to make adjustments to the prompts, the way in which structured output is handled, the retrieval settings, the routing rules, or the safety controls.

An evaluation harness proves especially useful here.

If the evaluation framework is well maintained, the team will be able to compare the current model and the replacement model using the same group of tasks prior to switching.

In the absence of it, the migration could become almost entirely subjective. Developers and users might try out a few examples, conclude that the new model “looks fine”, and only find out weeks later that it does not perform well on an important edge case.

Model changes should therefore be regarded as a standard aspect of AI system maintenance rather than as a rare occurrence.

Months 12 to 18: Expansion, Stabilization, or Decay

Usually, the system’s long-term direction becomes clearer after a year of production.

A successful deployment can be extended to other workflows. Since the infrastructure, the data pipelines, the monitoring system, the evaluation framework and the deployment process are already in place, some of the work needed for a second use case can be reused.

It doesn’t mean that the second use case will simply cost a fraction of the first; the extent of the savings will depend on how much of the existing architecture and domain knowledge in fact can be reused.

What is important is that the initial investment can provide a basis for future AI applications rather than simply serving as a one-off implementation.

The other option is a slow decline.

The person who is in charge changes to a different position. No one assumes responsibility for the evaluation set. The monitoring alerts are not read. Business processes are altered without the AI workflow being updated accordingly. The users come across so many errors that they begin to create manual workarounds.

It’s still possible that the system is online.

It could even display good uptime and response times.

In practice, however, it has been given up.

The distinction between a system that improves and one that gradually deteriorates is usually not so much a matter of the underlying model as it is of who is in charge. It is necessary to have someone responsible for keeping an eye on quality, examining failures, maintaining the evaluation data, and deciding when the system should be changed.

What Should “Full-Cycle” AI Development Actually Mean?

The term “full-cycle” appears frequently in AI development proposals, but it can mean very different things from one provider to another.

A meaningful full-cycle engagement should extend beyond deployment.

At minimum, the post-launch plan should answer several questions:

• Who monitors production quality?
• Which metrics are tracked, and how frequently are they reviewed?
• Who owns the evaluation set?
• How are new production failure cases added to testing?
• How will model or provider changes be evaluated?
• When will infrastructure and inference costs be reviewed?
• Who is responsible for incident response?
• What happens when the original development team hands the system over to internal engineering?
• How much ongoing engineering capacity is expected from the client?

Scopes for full-cycle AI software development that explicitly define these responsibilities provide a clearer picture of what happens after go-live.

That matters because AI maintenance is not simply fixing bugs. It is maintaining the relationship between the model, the data, the workflow, and the people using the system.

FAQ

How much should we budget for AI maintenance after launch?

There is not a single percentage that can be applied to all AI systems.

A good budget ought to include several categories such as quality monitoring, the maintenance of the evaluation set, infrastructure and inference costs, changes to the model, security updates, changes to the data pipeline, and continuous improvements to the workflow.

A fairly simple internal AI assistant will only need a small amount of engineering effort, while a high-volume or business-critical system involving complex integrations and human review will require considerably more.

Instead of using a fixed maintenance percentage, work out the cost of these activities separately and then check the budget once some production data are available.

What is model drift, and how fast does it happen?

“Model drift” is often used broadly, but several different changes can affect production performance.

Data drift occurs when the characteristics of incoming data change. Concept drift occurs when the relationship between inputs and expected outputs changes. There can also be changes in the underlying model or provider that affect system behavior.

There is no set timeline for drift; in some cases systems may stay stable for long periods while in others changes can occur quickly when there are alterations to products, processes, regulations, or document formats.

Which is why it is more useful to carry out continuous monitoring and regular evaluation than to assume that drift will take place in a certain month.

Do we need a full-time engineer to maintain an AI system?

Not necessarily.

The amount of capacity needed will vary according to the complexity of the system, how it is used, its importance to the business, and the speed at which it changes. For example, a small internal assistant might only need some engineering support from time to time, whereas a production system that is processing large numbers of transactions would require dedicated ownership.

The most important requirement is that there should be clear accountability.

Production quality, evaluation, changes to the model, monitoring, and escalation should all be the responsibility of one person. An AI system will be able to decline over time if that responsibility is not assigned to anyone.

The Takeaway

AI system maintenance starts as soon as the system interacts with real users.

The work goes beyond simply fixing bugs; teams also have to keep an eye on production quality, expand the evaluation data, respond to changes in the input patterns, optimize both model and infrastructure costs, and make preparations for changes in the underlying AI models.

The precise cost of maintenance will depend on the system in question. The engineering duty that has to be carried on continuously should not be underestimated.

Considering only the budget up to the go-live stage treats the deployment as if it is the end of the process.

For most production AI systems, it is the stage at which the more informative phase of the lifecycle starts.