Mcp_lab_tools

3 forks

2 stars

Free

What is mcp_lab_tools

mcp_lab_tools is a suite of virtual lab provisioning tools developed by nbritton, designed to facilitate the setup and management of virtual labs using Mirantis Cloud Platform (MCP). These tools are particularly useful for creating isolated environments that mimic real-world network configurations without the need for extensive physical infrastructure. By leveraging dummy networks and virtual machines, users can experiment with and deploy complex setups, enabling robust testing and development workflows.

How to Use mcp_lab_tools

To get started with mcp_lab_tools, you'll need an Ubuntu Server 18.04 LTS machine. Use the lab_setup.sh script to configure your server for virtual lab use. This script sets up various network interfaces, although you'll likely need to adjust the network settings to suit your specific needs. The script includes dummy networks that act like real interfaces but do not connect to the outside world, requiring a virtual routing gateway for Internet connectivity.

For network setup, you can use ClearOS for its straightforward OpenVPN implementation, or configure iptables for packet forwarding. The tool supports bonds, bridges, and VLANs on top of dummy interfaces, allowing you to replicate nearly any production network setup.

When creating virtual machines, use the create-virtual-machines script, modifying the prefix variable to differentiate between lab environments. This way, you can manage multiple virtual labs on a single or multiple physical machines without naming conflicts.

For high-availability (HA) MCP clusters in virtual labs, a minimum of 400GB of RAM is recommended. If unavailable, using an SSD as a swap partition can be a viable alternative. Stability improvements can be achieved by setting the processor's frequency governor to a fixed frequency and using a low latency kernel.

Key Features of mcp_lab_tools

Virtual Lab Environment: Enables the creation of isolated virtual labs on Ubuntu Server 18.04 LTS, perfect for testing and development.
Network Flexibility: Supports dummy networks, bonds, bridges, and VLANs, allowing users to mimic complex production network setups.
Resource Management: Facilitates multiple virtual labs through prefix modifications, enabling efficient resource allocation and naming conventions.
Virtual Routing: Integrates with ClearOS for OpenVPN support or utilizes iptables for network packet forwarding.
High Availability Setup: Supports HA MCP cluster setups, with recommendations for memory and storage optimizations, including SSD swap partitions and low latency kernels.
Snapshot and Rollback: Use ZFS or similar filesystems for atomic CoW snapshot capabilities, allowing easy rollback of entire lab environments.
Scalable Infrastructure: Supports scaling across multiple physical hypervisors with Open vSwitch and Stateless Transport Tunneling for creating layer 2 mesh overlay networks.

mcp_lab_tools provides a powerful and flexible solution for setting up virtual labs, enabling users to simulate and test complex network environments with ease. Whether you're developing, testing, or learning, these tools offer a practical approach to managing virtual infrastructures.

Mirantis MCP Virtual Lab Provisioning Tools Use lab_setup.sh to setup a Ubuntu Server 18.04 LTS machine for use as a virtual lab. The network configuration provided in the script for illustrative purposes, you will more then likely have to reconfigure the settings to the networks you want to use. The dummy networks are a key enabler here, they function as a virtual network interface and you can use them like you would any network interface... it's just that they're not connected to anything in the outside world. To enable connectivity between the Internet and the virtual networks you need a virtual routing gateway. I personally use ClearOS because it has a turnkey OpenVPN implementation (which allows you to VPN into the virtual networks from your MacBook via Tunnelblick). Alternatively you could probably also use iptables to forward packets to the right networks. Bonds, bridges, and VLANs are supported on top of the dummy virtual interfaces... which should allow you to replicate practically any configuration found in production. Also, the nested virtual machines running on the kvm nodes will require you to have a DHCP server running on the virtual deployment network, this is also why I use ClearOS rather then iptables. You can have more then one virtual lab, simply change the prefix variable before running the create-virtual-machines script. In a simple single physical machine layout you could simply do lab01-, lab02-, lab03-. Then when you run virsh list --all you will see all the machines, such as lab01-kvm01.mylab.local, lab02-kvm01.myotherlab.local. If you have multiple phsyical machines then I would recommend prefixing with something like aa- for hypervisor 1, ab- for hypervisor 2, and ac for hypervisor 3, etc... this would let you 52 physical hypervisor machines and unlimited subnets without having any naming collisions. Each discrete virtual lab should have its own virtual router, i.g. aa-vrouter, ab-vrouter. The setup of ClearOS can be automated for dynamic provisioning, I just haven't had time to play around with that. If you are running a full HA production scale MCP cluster in a virtual lab then you will need about a minimum of 400GB of system memory. If you don't have this much memory a workable alternative is to setup an SSD as a swap partition, preferably you'd have something like an Intel Optane 900p laying around, but a regular PCI NVMe SSD card should work well too. I've noticed during use that sometimes the virtual machines nested on the virtual kvm nodes can stall, for instance when running salt commands the process doesn't actually start and/or return. My educated guess is it's a problem with the process scheduler. It only happens under very heavy load, for instance on my Dell R920 with quad Intel Xeon E7-8895 v2 processors (60 physical cores, 120 hyper-threaded) this only happens after trying to run three full scale labs concurrently, which is equivalent to about 140 virtual machines. It could also be due to QPI bottlenecks because I was not doing any form of cpu pinning. I saw noticeable stability improvements by setting the processor's frequency governor to a fix frequency, dynamic turbo-boosting should be disabled. I also believe I noticed an improvement by using the lowlatency kernel, I recommend whatever the current HWE lowlatency kernel is: sudo apt-get install --install-recommends linux-lowlatency-hwe-18.04; It's best to setup the lab on ZFS, or another filesystem that supports atomic CoW snapshoting. This is fundamentally required because it lets you roll back the whole cluster with a a few simple commands. You would create each discrete lab in it's own seperate ZFS dataset, i.g. zpool create data raidz /dev/disk1 /dev/disk2 /dev/disk3; zfs create data/images; zfs set mountpoint=/var/lib/libvirt/images data/images; zfs create data/images/lab01; zfs snapshot data/images/lab01@my-snapshot-name; zfs rollback data/images/lab01@my-snapshot-name; You can also setup the virtual networks across machines, this is useful if you don't have enough resources on a single lab hypervisor instance to run the whole MCP HA stack. For instance if you have three machines with 128GB of memory on them each then this should be enough to memory to scale a production scale MCP cluster across them: lab hypervisor 1: kvm1, osd1/2, gtw1, and cmp1. lab hypervisor 2: kvm2, osd3/4, gtw2, and cmp2. lab hypervisor 3: kvm3, osd5/6, gtw3, and cmp3. The virtual dummy networks on the three lab hypervisors are all on separate layer 2 domains, which is to say that they're not connected at all except for layer 3 access through the vrouter gateway. The way you can connect the virtual dummy networks together on the three lab hypervisors into a single logical domain is to create a layer 2 mesh overlay network on the lab hypervisors. This can be done using Open vSwitch with Stateless Transport Tunneling ("STT"), I maintain a DPDK enabled OVS package repository for Ubuntu 18.04 here: