On-premise deployments for enterprises - things to consider.

Here are some key insights from my experience with on-premise deployment engagements for enterprise customers:

Enterprises enforce strict deployment standards, and vendors are expected to adhere to these requirements.

High Availability and Disaster Recovery

Clearly define HA and DR capabilities of the product upfront. This sets expectations and ensures compatibility with the customer's needs.

Using a stateless service design supports high availability. By ensuring service nodes do not retain session or state information across requests, the system can easily scale horizontally. This approach facilitates seamless load balancing to achieve high availability.

It common for enterprises to distribute their deployments across various regions, data centers, or availability zones to enhance disaster recovery capabilities. For stateful services that cannot concurrently run multiple instances (active-standby), it is crucial to establish a well-defined recovery strategy. This encompasses automatic/manual failover and failback procedures between active and standby instances, as well as implementing data and state replication methods (hot/warm/cold standby). Moreover, it's also crucial to have both short-term and long-term data backup solutions in place to mitigate the risk of data loss and ensure swift recovery when needed.

Sizing

Customers typically expect vendors to provide sizing estimations. One effective approach is to develop a sizing calculator that uses predefined inputs to estimate resource requirements based on anticipated usage patterns.

Estimating resource needs can be challenging if product usage patterns vary widely. However, every project must start somewhere. Therefore, it's prudent to begin with informed assumptions and contingencies, supported by robust monitoring strategies and a periodic review to make necessary adjustments.

Performance assurance

In an on-premises private cloud deployment setup, the customer's infrastructure can vary significantly in aspects such as virtualization technology, per-node resource limits, network bandwidth, and disk IO performance. Therefore, it's reasonable for them to expect the vendor to provide a comprehensive performance benchmarking setup, including test data and benchmarking scripts, to ensure confidence in the system's performance capabilities.

Observability

On-premise deployments present significant challenges for issue troubleshooting due to access restrictions. Therefore, it is essential to integrate robust monitoring and alerting capabilities to streamline issue investigation and resolution. Key aspects to consider include log aggregation, monitoring resource utilisation, tracking application metrics, setting thresholds, and configuring alerts for timely notifications when thresholds are exceeded.

Security and vulnerabilities

Enterprises typically enforce stringent security standards and expect vendors to comply accordingly. At minimum, the images/binaries intended for deployment on the customer's infrastructure must be free of any direct or third-party introduced security vulnerabilities.

Moreover, this requirement is ongoing. It is advisable to integrate vulnerability scanning mechanisms into the continuous integration (CI) process and regularly update third-party dependencies to their latest versions to incorporate necessary fixes and enhancements.

Licences

When utilizing third-party libraries, breaching licenses can result in legal disputes and significantly affect enterprises. Vendors are contractually obligated to bear the consequences of such violations, which can lead to substantial financial losses. Therefore, it is imperative to implement thorough checks to regularly scan for license violations when using third-party dependencies. It's highly recommended to integrate the vulnerability scans to the continuous integration pipeline to mitigate risks effectively.

Deployment automation

In case of private cloud deployments, customers generally maintain their own CI pipelines for deploying services and prefer integrating our services into these pipelines. In such cases, establishing a standardized automated process for sharing new product binaries is crucial. This process should include automated deployment scripts capable of deploying the binaries. Implementing such scripts ensures that we maintain control over deployment logic, which is essential for efficient maintenance in the future.

Project management

More than the above factors, engagements often fail due to inadequate project management and execution. Successful implementation requires a well-defined project plan with clear delivery milestones and associated timelines. It's crucial to document and obtain approval for the product architecture, deployment strategies, identified shortcomings, and underlying assumptions from stakeholders. Regular review and mitigation of project delays and risks should involve key stakeholders from the customer side.

I hope these insights will help someone involved in implementing products for enterprise customers. I invite you to share your experiences and insights, and to highlight any additional aspects you believe are crucial for successful deployments in enterprise environments.