Autodeploy Image: An error occurred while generating the image [Entry is too large to be added]

This issue occurred with below environment:

vcsa 6.5, embedded vum,autodeploy and image builder service enabled.

  • For image customization we mapped more images [ approx 2 GB] so it was not allowing to map new image, and throwing below error message
Error while Autodeploy Image... An error occurred while generating the image : Error : An error occurred while performing the task Entry is too large to be added to cache, please remove any imported depots you are not using...
  • Delete the images, which are not in-use, or apply the below workaround [increase the cacheSize]
cat /etc/vmware-imagebuilder/sca-config/imagebuilder-config.propsloglevel=INFO
vmomiPort=8098
httpPort=8099
cacheSize_GB=4

ls -lh /storage/imagebuilder/exports/
total 361M

Go in vcsa --> Administrator-->System Configuration-->Services

Restart Auto Deploy

Restart ImageBuilder Service

 

 

How to set vrops [vrealize operation manager] forgotten root password

Recently, in my testing environment i forgot the root password. I did the following step to reset password:

-Restart the vrops node

-Edit the Boot option “init=/bin/bash”

Boot Options vga=0x311 elevator=noop noexec=on nousb audit=1 init=/bin/bash

-Once system boots, type below command:

"passwd root"

It will prompt for new password. Provide the new password and reboot the system!!!

pyVmomi module: Script for fetching hardware information from ESXi

import argparse
from pyVmomi import vim
from pyVim.connect import SmartConnect,Disconnect
import atexit
import ssl

def validate_options():
  parser = argparse.ArgumentParser(description='input parameters')
  parser.add_argument('-d','--dest_host',dest='dhost',required=True,help='The ESxi destination host IP')
  parser.add_argument('-v','--vc_host',dest='vchost',required=False,help='The VC ip')
  parser.add_argument('-u','--vc_user',dest='vcuser',required=True,help='VC username')
  parser.add_argument('-p','--vc_pass',dest='vcpasswd',required=True,help='VC passwd')
  args = parser.parse_args()
  return args

def getHostID(content,dhost):
  if content.searchIndex.FindByIp(None,dhost,False):
    host = content.searchIndex.FindByIp(None,dhost,False)
  else:
    host = content.searchIndex.FindByDnsName(None,dhost,False)
  return host

def get_HostInfo(content,dhost):
   search_index = content.searchIndex
   root_folder =  content.rootFolder
   view_ref = content.viewManager.CreateContainerView(container=root_folder,type=[vim.HostSystem], recursive=True)
   host = view_ref.view[0]
   #print host.name
   print 'UUID INFO %s' %(host.summary.hardware.uuid)
   print 'Hardware Model %s' %(host.summary.hardware.model)
   print '%s Server has %s Biosversion'%(host.hardware.biosInfo.vendor,host.hardware.biosInfo.biosVersion)
   pcilist=host.hardware.pciDevice
   print '{0}'.format("Vendor Name").ljust(20)+ '{0}'.format("Device Name").ljust(120)+ '{0}'.format("Slot").ljust(30)+ '{0}'.format('Device ID').ljust(10)
   print '*************************************************************************************************************************************************************************************************'
   for i in pcilist:
     a = i.vendorName
     b = i.deviceName
     c = i.deviceId
     d = i.slot
     print '{0}'.format(a).ljust(20)+ '{0}'.format(b).ljust(120)+ '{0}'.format(d).ljust(30)+ '{0}'.format(c).ljust(10)
     #print '%s has devicename %s and  device ID %s'%(i.vendorName,i.deviceName,i.deviceId)
   print '*********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************' 
def main():
  opts = validate_options()
  if opts.vchost:
    print 'Connecting to vcenter and collecting sensor info for %s' %opts.dhost
  else:
    print 'Connecting to esxi host for %s' %opts.dhost
    opts.vchost = opts.dhost
  si = SmartConnect(host=opts.vchost, user=opts.vcuser, pwd=opts.vcpasswd)
  content = si.RetrieveContent()
  #print content
  hostinfo = get_HostInfo(content,opts.dhost)
  hostid = getHostID(content,opts.dhost)
  sensorinfo=hostid.runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo
  print '{0}'.format("Sensor").ljust(30)+ '{0}'.format("Sensor Detail").ljust(90)+ '{0}'.format('Status').ljust(10)+ '{0}'.format('Reading').ljust(10) +'{0}'.format('Units').ljust(13)+ '{0}'.format('Summary').ljust(20)
  print '**************************************************************************************************************************************************************************************************************'
  for i in sensorinfo:
    j = i.healthState
    a=str(i.currentReading)
    b=i.baseUnits
    c=i.sensorType
    print '{0}'.format(c).ljust(30)+ '{0}'.format(i.name).ljust(90)+ '{0}'.format(j.label).ljust(10)+ '{0}'.format(a).ljust(10) + '{0}'.format(b).ljust(13)+ '{0}'.format(j.summary).ljust(20)

if __name__ =='__main__':
  main()
How to run this script : 

python <name of script> -v <vc server> -d <esxi host which hardware want to list> -u <vc user name> -p <vc password>

This script is written in python. I have used the  pyVmomi module.

vSphere On-disk Metadata Analyzer (VOMA)

VOMA helps in performing VMFS file system metadata checks. This utility scans the VMFS volume metadata and highlights any inconsistencies.

VOMA provides four modules, and except for lvm, each of them has a fix function:

lvm-  Checks datastore’s logical device header,logical volume header and physical extent mapping

  • vmfs – This module checks vmfs hearder,resource file, heartbeat region, file descriptor ,connectivity etc.0
  • ptbl –  Module checks the partition table and provide table structure.
    • Phase 1: Checking device for valid primary GPT
    • Phase 2: Checking device for a valid backup GPT
    • Phase 3: Checking device for valid MBR table
    • Phase 4: Searching for valid file system headers
voma [OPTIONS] -m module -d device

-m, --module      Name of the module to run.

                    Available Modules are

                      1. lvm

                      2. vmfs

                      3. ptbl

-f, --func        Function(s) to be done by the module.

                     Options are

                       query   - list functions supported by module

                       check   - check for Errors

                       fix     - check & fix

                       dump    - collect metadata dump

-d, --device      Device/Disk to be used

-s, --logfile     Path to file, redirects the output to given file

-x, --extractDump Extract the dump collected using VOMA

-D, --dumpfile    Dump file to save the metadata dump collected

-v, --version     Prints voma version and exit.

-h, --help        Print this help message.
Example:

voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:x

voma -m vmfs -f dump -d /vmfs/devices/disks/naa.xxxx:x -D dumpfilename
voma -m vmfs -f check -d /vmfs/devices/disks/<device-id>

Checking if device is actively used by other hosts

Initializing VMFS Checker..|Scanning for VMFS-3/VMFS-5 host activity (512 bytes/HB, 2048 HBs).

Found 3  actively heartbeating hosts on device '/vmfs/devices/disks/<device id>

1): MAC address

2): MAC address

3): MAC address
voma -m ptbl -f check -d /vmfs/devices/disks/<device id>

Running Partition table checker version 0.1 in check mode

Phase 1: Checking device for valid primary GPT

Phase 2: Checking device for a valid backup GPT

Phase 3: Checking device for valid MBR table

Phase 4: Searching for valid file system headers

No valid LVM headers detected

pyVmomi rpm for centos7

pyVmomi is the Python SDK for the VMware vSphere API that allows you to manage ESX, ESXi, and vCenter. pyVmomi is available on git.

https://github.com/vmware/pyvmomi

I have created a rpm format of same pyVmomi SDK for centos7.  This rpm will be installed in the /opt folder on your centos 7.

Below is the Spec file:

%define BUILD pyvmomi_master.1.0.1.x86_64
Summary: Pyvmomi package
Name: pyvmomi_master
Release: 1.0
Version: 1
License: Apache License 2.0
Requires: python-six
Requires: python-requests
Requires: python-setuptools
BuildArch: noarch

%description
This package contains the vSphere python SDK

%post
%files
%defattr(-,root,root,-)
/opt/pyvmomi-master
%doc
%changeLog
* Fri Jul 14 2017 Amit <amit@openwriteup.com> 1-1.0
- Pyvmomi 6.5

Once you install the rpm, it will be in the /opt/pyvmomi-master folder.

 rpm -ivh pyvmomi_master-1-1.0.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:pyvmomi_master-1-1.0             ################################# [100%]


################################# [100%]
[root@devbox noarch]# ls /opt/pyvmomi-master/
docs  LICENSE.txt  MANIFEST.in  NOTICE.txt  pyVim  pyVmomi  README.rst  requirements.txt  sample  setup.cfg  setup.py  test-requirements.txt  tests  tox.ini

Post installation of the package, we need to run following step:

[root@devbox pyvmomi-master]# python setup.py –help
Common commands: (see ‘–help-commands’ for more)

setup.py build      will build the package underneath ‘build/’
setup.py install    will install the package

 python setup.py install
running install
running bdist_egg
running egg_info
creating pyvmomi.egg-info
writing requirements to pyvmomi.egg-info/requires.txt
writing pyvmomi.egg-info/PKG-INFO
writing top-level names to pyvmomi.egg-info/top_level.txt
writing dependency_links to pyvmomi.egg-info/dependency_links.txt
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'

What’s New in vSphere 6.5: vCenter management clients

vSphere 6.5 VMware introduced html5 support.

In older release VMware was providing two types of clients:

– vSphere Client [ exe installer]

– vSphere Web client [flash based]

vSphere Client, from vSphere 5.5 onwards started  providing lot of restriction. VMware stopped latest virtual hardware vendor support, other core feature configuration from vSphere client.

In case of vSphere Web client [flash based], its performance was not up to mark to handle big environment. This flash based solution has lot of performance issues.

In vSphere 6.5, VMware has two types of clients:

HTML5 [vSphere client]

Flash [vSphere Web Client]

HTML5 [vSphere client]:VMware agrees that Flash is not the solution for the long-term. Our long-term direction is to utilize HTML5. In vSphere 6.5, we have released a supported version of an HTML5 based web client which we call “vSphere Client”. The vSphere Client is part of the vCenter Server (both appliance and Windows) and is configured to work out of the box.

Access Url:  https://<ip or fqdn of VC>/ui

Note:This HTML5 based client was originally released as a fling back in March 2016 and has been releasing a new version every week.

https://labs.vmware.com/flings/vsphere-html5-web-client#instructions

vSphere Web Client: The vSphere Client (HTML5) released in vSphere 6.5 has a subset of features of the vSphere Web Client (Flash/Flex). Until the vSphere Client achieves feature parity, we might continue to enhance and/or add new features to vSphere Web Client.

https://blogs.vmware.com/vsphere/2016/12/new-vcenter-management-clients-vsphere-6-5.html

Cloud Foundry

Cloud Foundry is an open platform as a service, providing a choice of clouds, developer frameworks and application services.

In cloud era, the application platform will be delivered as a service, often described as Platform as a Service (PaaS).Cloud Foundry is an open source project and is available through a variety of private cloud distributions and public cloud instances.

Cloud native stack , this is layered stack. Below diagram shows the cloud native stack of cloud foundry.

Cloud native stack

Its a stack, which provides your software  a environment to run. This layer provides a platform to run developer software. This is four layer stack.

Infrastructure layer [ IAAS]: This enables the complete stack. This layer provides resource. This layer can be aws,vmware vSphere,vCloud air or openstack or Microsoft azure. The kind of operation this layer provides :Provision a server, install vm on server, install os on the vm and operations on the vm [ start,stop]. This layer automate all the operation.Basically all the vendor provides the api to automate the complete IAAS layer operations. This API provide is cloud provide interface [CPI]

Infrastructure Automation: This layer takes care of CPI automation. In case of cloud foundry Bosh take care of automating Cloud provider interface. This will take care of provisioning of vm, creating of database vm,patching,upgrading or high availability etc, This layer will automate all the infrastructure operations task. In case of cloud foundry, BOSH automates all the infra task. If we package our software and provides to BOSH,  BOSH will take care of all the provisioning and configuration

RUNTIME Layer: This is cloud foundry layer. This layer is also called Elastic run time layer as well. This layer containerized the application. It takes care of Domain,routing and complete orchestration. This layer of take care of all the the orchestration task. All kind of scaling can be also orchestrated in this layer.

Application Layer: In this layer all the programming languages comes. This provides the environment for programming languages. This not only provides the environment also provide the supported library as well. It also contains lot of middleware,databases . As developer, need to take care of application rest of the stuff will be take are.

Just like aws lamda, it only provide the environment to developer. You need not to worry about which OS its running, what networking etc. Every thing is fully orchestrated. It has scaling , HA kind of features as well…

This link provides details : https://github.com/cloudfoundry

 

 

 

vSphere SDK for python : pyVmomi

VMware has released pyVmomi for developers who wants to code in python. pyVmomi is a VMware SDK to manage ESXi and vCenter using python.
Using pip we can install:
pip install pyvmomi

Developers can download the pyvmomi package from python.org

https://pypi.python.org/pypi/pyvmomi

When we extract and list the package
LICENSE.txt NOTICE.txt pyVim pyvmomi.egg-info requirements.txt setup.cfg test-requirements.txt tox.ini MANIFEST.in PKG-INFO pyVmomi README.rst sample setup.py tests

Perform the below steps:
 python setup.py install
 python setup.py develop

pyVmomi package provide mainly two modules : pyVim and pyVmomi
Let’s explore pyVim. This module will be used to connect to ESXi and vSphere Center.
>>> import pyVim
>>> help(pyVim)
PACKAGE CONTENTS
connect
task
>>>import pyVim.connect
NAME
pyVim.connect – Connect to a VMOMI ServiceInstance.
>>>help(pyVim.connect)
This contents two function to connect service instance: SmartConnect and Connect
SmartConnect(protocol=’https’, host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, path=’/sdk’, preferredApiVersions=None, keyFile=None, certFile=N
one, thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Determine the most preferred API version supported by the specified server,then connect to the specified server using that API version, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access.Clients should modify the service parameter only when connecting to a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine.

Connect(host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, adapter=’SOAP’, namespace=None, path=’/sdk’, version=None, keyFile=None, certFile=None,
thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Connect to the specified server, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access. Clients should modify the service parameter only when connecting to
a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine

 

We can connect using SmartConnect
From pyVim.connect import SmartConnect
conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn

'vim.ServiceInstance:ServiceInstance'

This is service instance type of managed object. For writing code further we can make use of vSphere mob interface: http://<vc-ip>/mob

mob-2

Wrote a small code which fetch the build number and name of datecenter

#!/bin/python
from pyVim.connect import SmartConnect
conn = SmartConnect(host="<VC hostname>",user="<username>",pwd="<password>")
print ("Multi host is supported")
print (conn.capability.multiHostSupported)
print ("----------------------------------------------------------")
print ("VC System Current time")
print (conn.CurrentTime())
print ("----------------------------------------------------------")
print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (conn.content.about.build)
print ("----------------------------------------------------------")
datacenter = conn.content.rootFolder.childEntity
print ("Name of datacenters")
print ("----------------------------------------------------------")
for i in datacenter:
 print (i.name)
# python vcconnect.py 
Multi host is supported
True
----------------------------------------------------------
VC System Current time
2016-09-22 14:33:17.
----------------------------------------------------------
VC server build info
6.0.0
VMware vCenter Server 6.0.0 build-xxxxxx
----------------------------------------------------------
Name of datacenters
----------------------------------------------------------
test_Lab

Below are the screenshot how I traverse for Version number:

conn.content.about.version

conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn

'vim.ServiceInstance:ServiceInstance'

conn is Service Instance Object. listing the properties and methods from mob interface.

mob-3

In properties segment, I am listing the content property.This content the about property.

mob-4

In about we can list version:

mob-5

That’s the way below code has traversed.

print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (conn.content.about.build)

 

Vmware VRO (VCO) :How to design workflow :Basic Workflow[series3]

In this section, We will try to write a workflow from scratch… When we start writing any workflow, we need to start from following:

  • What needs to be developed, means understand the requirement properly
  • Perform manually what parameters are require, possibly from virtual center.

Lets start for a workflow, where we are get virtual machine snapshots. When we do manually, we connect to esx(i) host (using VIC/web client), for a particular vm right click and snapshot manager.

Lets have a api search in vRo, searching

snapshot2

In Api search, we will find lot snapshot related stuff such as revert,create.. We need to select what we are looking for “get vm snapshot”.

snapshot3

There are methods or attribute define in different object. In VcPlugin, getAllVirtualMachineSnapshots method is available. Lets have a look into this method:

snapshot

This method has two parameters and one return type value.

  • Parameters
    • additonalPropertyFilters(array of String)
    • query(string) /*Xpath query*/
    • Return Type: Array of VcVirtualMachineSnapshot

First parameter additional property filters, its a big topic to discuss, “PropertyFilter is actually applying a filter to each result, as in, return the following fields for each result instead of filtering the results set based on the PropertyFilter.  So it will return the properties as the result of filter.

Xpath query will be used to search the result from vCo side.

Lets start writing the workflow, we create a workflow and go in schema , we see the below figure.

snapshot1\

In this workflow we need the list of virtual Machines which snapshots need to be searched. So first input is “List of Virtual Machines: So our input is Vc:VirtualMachine array type. Output is “Array of VcVirtualMachineSnapshot”. So we define Snapshot output type VC:VirtualMachineSnapshot.

Currently in schema we select the scriptable task.

snapshot5

Visual binding of the inputs and output in schema:

snapshot6

The Scripting part :

for(i in VMs) {
var vmpattern = VMs[i];
var XPath = "xpath://name[starts-with(.,'"+vmpattern+"')]";
ListSnapshot = VcPlugin.getAllVirtualMachineSnapshots(null,XPath);
Snapshot=ListSnapshot;
System.log (Snapshot);
}

Above is scripting part, which is working state,tried to use all the parameters defined above..

Below is the presentation layer, when we start executing the workflow, this is the layer will give the front end. It will ask the list of the array which we need

snapshot7

snapshot4

In upcoming series, we will try to understand more about propertyfilter… it will help for further coding…

 

Amazon cloud vm : How to access in secure environment

Recently, One of my client they were doing the proof of concept for amazon aws. Till Now, I just used for trial purpose, took a single ec2 instance and assigned a public ip, was able to access easily using putty. Just download the .pem file and access using ssh by any linux box.

In my client case they were using vpc (virtual private cloud), in that they have defined the availability zone (subnet are defined). When I was selecting any vpc, it has shows mapped availability zone. Client has provided one jump server as well from where we will be doing ssh or can use aws command line to control the environment. The complete environment is access aws private ip ,since it has mapped to it.

Problem faced and applied solution:

During the phase of starting, initially i have created ec2 instance in my assigned vpc, when i was trying to access it was not pinging from jump box. After googling a lot I found that “Security group” which i am using, doesn’t mention Source ip range from which it has to communicate.

I created a new security group  mentioning source “0.0.0.0/0” but it has started giving the security warning, so i went back and created the correct range “192.168.0.0/32”. After that i was able to ping my ec2 instance from my jump server.

Next step i created a key pair and downloaded the .pem file, while i was using that .pem file in my environment somehow it was not able to do ssh from the jump server. I was able to ping the aws instance but not able to connect.

I have moved the .pem file to file format: mv <file.pem> <file>

ssh <file option>  <file> ec2-user@<instance private ip>

now i was running aws command from the jump server (which is already configured on the system)

aws ec2 describe-instances

It was giving error not configured. Tried the below command

aws configure

This command ask “Access key id”, “Secret access key”,”default region”, and “output format”.

Access key id and secret access key information will get from IAM service configuration (Identity and Access Management), In users segment select your user. In that section you can create access key and activate it. Download it, It has both the information access key and secret.

Region information you can find from aws web page, which region you have selected, and output format (text,json or xml..) , once you ready with this information, please use the same command “aws configure”, will able to configure. Now if you run any command ” aws help” It will work from command line…

It was overall a good experience, yet to have lot of information to fetch…will definitely share