Boto 3: Basic and setup

Boto : Boto is a SDK designed to improve the use of the python programming language in aws.

Setup requirement:

  • Aws signup
  • Python version 2.7
  • Pycharm 3.3 [ide]
  • Pip setup

For programmatic access: We need to enable the access in the aws iam:

For programmatically user access , secret key and access id.

Setup awscli on windows

  • Awscli
  • Prerequires:
    • Check your system has Python 2.7
    • Pip is configured

Configure awscli

Configure the access key and id

C:\Users\amitm>aws configure

AWS Access Key ID [****************Z2FA]:

AWS Secret Access Key [****************V270]:

Default region name [ap-south-1]:

Default output format [test]:

Check setup?

C:\Users\amitm>aws s3 ls

2018-10-15 13:08:20 cf-templates-106h68kzl5m34-us-east-2

2018-11-08 23:39:48 openwriteup

2018-11-08 23:46:12 openwriteup-1

2018-11-09 00:16:44 test-openwriteup

What is awscli??

  • This is a command line tool
  • If we are writing script we can use it
  • Testing purpose or want to use shell or powershell it is useful that

Setup boto3

  • Pip install boto3
  • Test boto3
    • Python
    • Import boto3
    • Help(boto3)


  • A low-level interface to a growing number of Amazon Web Services. The botocore package is the foundation for the AWS CLIas well as boto3.
  • Botocore provides the low level clients, session, and credential & configuration data. Boto 3 builds on top of Botocore by providing its own session, resources and collections.
  • botocore does not provide higher-level abstractions on top of these services, operations and responses. That is left to the application layer. The goal of botocore is to handle all of the low-level details of making requests and getting results from a service

Core concepts of boto3


  • higher-level, object-oriented API
  • generated from resource description
  • uses identifiers and attributes
  • has actions (operations on resources)
  • exposes subresources and collections


import boto3

s3 = boto3.resource('s3')

bucket = s3.Bucket('mybucket')

for obj in bucket.objects.all():

print(obj.key, obj.last_modified)

Boto Client:

  • low-level service access
  • generated from service description
  • exposes botocore client to the developer
  • typically maps 1:1 with the service API
  • snake-cased method names (e.g. ListBuckets API => list_buckets method)


import boto3

client = boto3.client('s3')

response = client.list_objects(Bucket='mybucket')

for content in response['Contents']:

obj_dict = client.get_object(Bucket='mybucket', Key=content['Key'])

print(content['Key'], obj_dict['LastModified'])


Difference Between resource and client:

Resource object is very high level object, every operation with resource object would be high level operation. We may not have all the operation with resource.

Client is low level object, so whatever operation we want to perform its always be available. Client operations are mostly dictionary operation.


  • stores configuration information (primarily credentials and selected region)
  • allows you to create service clients and resources

Simple object to get it connected to particular aws account or iam account. If i want to connect any iam acocunt, session object will be used.


  • Automatically handles pagination
  • Yields individual pages
  • You must process each pages

Example: I have three thousand object in my s3 bucket, which i want to list. Boto3 Api can only list till a limit (1000 object). In such cases paginator can be used to list all the 3k objects. It will be using 3 pages to list .


Waiter are used for reach waiting to reach certain state

Example: I have ec2 instance, which i newly launched, it takes some time to reach running state. For that purpose we can use waiter


VMware VCSA 6.x backup using ansible

Recently, I have one project requirement where i have to use vmware rest api and ansible to take the vcsa backup. In this blog we will first explore vcsa apiexplorer and traverse to rest api for backup

we need to open: https:<vcsa fqdn or ip>/apiexplorer

Select the appliance from the drop down and traverse to the backup job

explore POST /appliance/recover/backup/job . It will have parameter values

These are the option we will be using while coding in ansible. parts option you can get from vcsa vami console.

SEAT is optional.

After exploring api, its time to look into ansible. First part would be login to vcsa apiexplore. This require authentication.

VCSA login using rest api /ansible code snip

As explain above for backup request body, same way for ansible we need write json file

Now we need to write the backup task in ansible yaml file

So backup.yaml will contain login and backup task. Backup task will be calling json file , which contain vcsa backup config info.

HPONCFG : To delete local user and add AD user

New requirement came for ilo3 and il04 for hpservers, below are the requirements:

-Remove local user

-Add Ad user.

We can add it manually, but the requirement for huge setup of server. After searching, I found hponcfg tool, which runs through xml. Below is the reference link, This links has all the xml files

Delete local user xml. We need to run this xml with hpocnfg tool (which is available for windows version as well)






Check VMware VCSA certs validity using Ansible and RestAPI

Recently a requirement came, where VMware VCSA 6.x compliance need to be checked using Ansible.  Vendor wants to use VMware RestApi [Not interested to use VMware Python SDK].

Note: VMware Ansible module comes with VMware Python SDK [PyVmomi]. This compliance check was having

ssh , ipv6, ntp, CA certs, DNS check and set (in case of value has changed).  Most of the option was available except CA cert check. Below yaml code has three parts;

  • VCSA login
  • cert fetch
  • logging


This is used for Product hardening!!


pyVmomi module: Script for fetching hardware information from ESXi

import argparse
from pyVmomi import vim
from pyVim.connect import SmartConnect,Disconnect
import atexit
import ssl

def validate_options():
  parser = argparse.ArgumentParser(description='input parameters')
  parser.add_argument('-d','--dest_host',dest='dhost',required=True,help='The ESxi destination host IP')
  parser.add_argument('-v','--vc_host',dest='vchost',required=False,help='The VC ip')
  parser.add_argument('-u','--vc_user',dest='vcuser',required=True,help='VC username')
  parser.add_argument('-p','--vc_pass',dest='vcpasswd',required=True,help='VC passwd')
  args = parser.parse_args()
  return args

def getHostID(content,dhost):
  if content.searchIndex.FindByIp(None,dhost,False):
    host = content.searchIndex.FindByIp(None,dhost,False)
    host = content.searchIndex.FindByDnsName(None,dhost,False)
  return host

def get_HostInfo(content,dhost):
   search_index = content.searchIndex
   root_folder =  content.rootFolder
   view_ref = content.viewManager.CreateContainerView(container=root_folder,type=[vim.HostSystem], recursive=True)
   host = view_ref.view[0]
   print 'UUID INFO %s' %(host.summary.hardware.uuid)
   print 'Hardware Model %s' %(host.summary.hardware.model)
   print '%s Server has %s Biosversion'%(host.hardware.biosInfo.vendor,host.hardware.biosInfo.biosVersion)
   print '{0}'.format("Vendor Name").ljust(20)+ '{0}'.format("Device Name").ljust(120)+ '{0}'.format("Slot").ljust(30)+ '{0}'.format('Device ID').ljust(10)
   print '*************************************************************************************************************************************************************************************************'
   for i in pcilist:
     a = i.vendorName
     b = i.deviceName
     c = i.deviceId
     d = i.slot
     print '{0}'.format(a).ljust(20)+ '{0}'.format(b).ljust(120)+ '{0}'.format(d).ljust(30)+ '{0}'.format(c).ljust(10)
     #print '%s has devicename %s and  device ID %s'%(i.vendorName,i.deviceName,i.deviceId)
   print '*********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************' 
def main():
  opts = validate_options()
  if opts.vchost:
    print 'Connecting to vcenter and collecting sensor info for %s' %opts.dhost
    print 'Connecting to esxi host for %s' %opts.dhost
    opts.vchost = opts.dhost
  si = SmartConnect(host=opts.vchost, user=opts.vcuser, pwd=opts.vcpasswd)
  content = si.RetrieveContent()
  #print content
  hostinfo = get_HostInfo(content,opts.dhost)
  hostid = getHostID(content,opts.dhost)
  print '{0}'.format("Sensor").ljust(30)+ '{0}'.format("Sensor Detail").ljust(90)+ '{0}'.format('Status').ljust(10)+ '{0}'.format('Reading').ljust(10) +'{0}'.format('Units').ljust(13)+ '{0}'.format('Summary').ljust(20)
  print '**************************************************************************************************************************************************************************************************************'
  for i in sensorinfo:
    j = i.healthState
    print '{0}'.format(c).ljust(30)+ '{0}'.format( '{0}'.format(j.label).ljust(10)+ '{0}'.format(a).ljust(10) + '{0}'.format(b).ljust(13)+ '{0}'.format(j.summary).ljust(20)

if __name__ =='__main__':
How to run this script : 

python <name of script> -v <vc server> -d <esxi host which hardware want to list> -u <vc user name> -p <vc password>

This script is written in python. I have used the  pyVmomi module.

pyVmomi rpm for centos7

pyVmomi is the Python SDK for the VMware vSphere API that allows you to manage ESX, ESXi, and vCenter. pyVmomi is available on git.

I have created a rpm format of same pyVmomi SDK for centos7.  This rpm will be installed in the /opt folder on your centos 7.

Below is the Spec file:

%define BUILD pyvmomi_master.1.0.1.x86_64
Summary: Pyvmomi package
Name: pyvmomi_master
Release: 1.0
Version: 1
License: Apache License 2.0
Requires: python-six
Requires: python-requests
Requires: python-setuptools
BuildArch: noarch

This package contains the vSphere python SDK

* Fri Jul 14 2017 Amit <> 1-1.0
- Pyvmomi 6.5

Once you install the rpm, it will be in the /opt/pyvmomi-master folder.

 rpm -ivh pyvmomi_master-1-1.0.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:pyvmomi_master-1-1.0             ################################# [100%]

################################# [100%]
[root@devbox noarch]# ls /opt/pyvmomi-master/
docs  LICENSE.txt  NOTICE.txt  pyVim  pyVmomi  README.rst  requirements.txt  sample  setup.cfg  test-requirements.txt  tests  tox.ini

Post installation of the package, we need to run following step:

[root@devbox pyvmomi-master]# python –help
Common commands: (see ‘–help-commands’ for more) build      will build the package underneath ‘build/’ install    will install the package

 python install
running install
running bdist_egg
running egg_info
creating pyvmomi.egg-info
writing requirements to pyvmomi.egg-info/requires.txt
writing pyvmomi.egg-info/PKG-INFO
writing top-level names to pyvmomi.egg-info/top_level.txt
writing dependency_links to pyvmomi.egg-info/dependency_links.txt
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest template ''
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'

vSphere SDK for python : pyVmomi

VMware has released pyVmomi for developers who wants to code in python. pyVmomi is a VMware SDK to manage ESXi and vCenter using python.
Using pip we can install:
pip install pyvmomi

Developers can download the pyvmomi package from

When we extract and list the package
LICENSE.txt NOTICE.txt pyVim pyvmomi.egg-info requirements.txt setup.cfg test-requirements.txt tox.ini PKG-INFO pyVmomi README.rst sample tests

Perform the below steps:
 python install
 python develop

pyVmomi package provide mainly two modules : pyVim and pyVmomi
Let’s explore pyVim. This module will be used to connect to ESXi and vSphere Center.
>>> import pyVim
>>> help(pyVim)
>>>import pyVim.connect
pyVim.connect – Connect to a VMOMI ServiceInstance.
This contents two function to connect service instance: SmartConnect and Connect
SmartConnect(protocol=’https’, host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, path=’/sdk’, preferredApiVersions=None, keyFile=None, certFile=N
one, thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Determine the most preferred API version supported by the specified server,then connect to the specified server using that API version, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access.Clients should modify the service parameter only when connecting to a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine.

Connect(host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, adapter=’SOAP’, namespace=None, path=’/sdk’, version=None, keyFile=None, certFile=None,
thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Connect to the specified server, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access. Clients should modify the service parameter only when connecting to
a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine


We can connect using SmartConnect
From pyVim.connect import SmartConnect
conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn


This is service instance type of managed object. For writing code further we can make use of vSphere mob interface: http://<vc-ip>/mob


Wrote a small code which fetch the build number and name of datecenter

from pyVim.connect import SmartConnect
conn = SmartConnect(host="<VC hostname>",user="<username>",pwd="<password>")
print ("Multi host is supported")
print (conn.capability.multiHostSupported)
print ("----------------------------------------------------------")
print ("VC System Current time")
print (conn.CurrentTime())
print ("----------------------------------------------------------")
print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (
print ("----------------------------------------------------------")
datacenter = conn.content.rootFolder.childEntity
print ("Name of datacenters")
print ("----------------------------------------------------------")
for i in datacenter:
 print (
# python 
Multi host is supported
VC System Current time
2016-09-22 14:33:17.
VC server build info
VMware vCenter Server 6.0.0 build-xxxxxx
Name of datacenters

Below are the screenshot how I traverse for Version number:


conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn


conn is Service Instance Object. listing the properties and methods from mob interface.


In properties segment, I am listing the content property.This content the about property.


In about we can list version:


That’s the way below code has traversed.

print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (


List all the running instance on amazon VPC

In my test environment, I have amazon VPC, which I am accessing using Linux server.

For performing all the activities on amazon vpc, I have used python script.For automation in amazon VPC, aws provides module boto3, which need to be installed using python pip. Using this module we list all the running instances.In below script I am creating a config file and then reading that config file.Following script perform all these steps:

  • Create a config file, which is required to connect aws vpc.
  • Read the config file, and list all the instance following details:
    • ‘Name’:name,
    • ‘Type’: instance.instance_type,
    • ‘State’:instance.state[‘Name’],
    • ‘Private IP’:instance.private_ip_address,
    • ‘Public IP’: instance.public_ip_address,
    • ‘Launch Time’: instance.launch_time



import ConfigParser,boto3,os,sys,paramiko
from collections import defaultdict
config = ConfigParser.RawConfigParser()
#When adding sections or items, add them in reverse order
config.set('USER','AWS_Profile','<aws user to login>')
config.set('EC2','Region','<aws region>')

#Writing configruation to config file
name = raw_input("Enter the config file name::: ")
with open(name, 'wb') as configfile:
#Reading the config file
config1 = ConfigParser.ConfigParser()
ses = boto3.Session(profile_name = config1.get("USER", "AWS_Profile"))
ec2 = ses.resource('ec2')
key = paramiko.rsakey.RSAKey.from_private_key_file(filename=config1.get("USER","Private_Key")) 
running_instances = ec2.instances.filter(Filters=[{
'Name': 'instance-state-name',
'Values': ['running']}])
ec2info = defaultdict()
for instance in running_instances:
for tag in instance.tags:
if 'Name' in tag['Key']:
name = tag['Value']
ec2info[] = {
'Type': instance.instance_type,
'Private IP':instance.private_ip_address,
'Public IP': instance.public_ip_address,
'Launch Time': instance.launch_time
attributes = ['Name','Type','State','Private IP','Public IP','Launch Time']
for instance_id, instance in ec2info.items():
for key in attributes:


output of the script :

Private IP:
Public IP:None
Launch Time:2016-08-26 23:09:17+00:00



aws instance listing using python sdk

This blog is for those, who are very new to aws and python. They want to start both of them together. Assuming they have setup boto3 environment in their test lab.

In lab setup type python: python

It will give python prompt, we can explore boto3.

>>> import boto3
>>> dir(boto3)
[‘DEFAULT_SESSION’, ‘NullHandler’, ‘Session’, ‘__author__’, ‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘__version__’, ‘_get_default_session’, ‘client’, ‘docs’, ‘exceptions’, ‘logging’, ‘resource’, ‘resources’, ‘session’, ‘set_stream_logger’, ‘setup_default_session’, ‘utils’]

Perform help (boto3) ,It will show the package content with this package..

docs (package)
dynamodb (package)
ec2 (package)
resources (package)
s3 (package)

Lets import the resources : from boto3 import resources

>>> dir (boto3.resources)
[‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘action’, ‘base’, ‘collection’, ‘factory’, ‘model’, ‘params’, ‘response’]



/*perform following on your python console or write a .py script
import boto3
#help(ec2) /*it will list all the available option with ec2*/
#help(ec2.instances) /*search for filter*/
#help(ec2.instances.filter /*list the filter option and list syntax
/* instance_iterator = ec2.instances.filter(
 | DryRun=True|False,
 | InstanceIds=[
 | 'string',
 | ],
 | Filters=[
 | {
 | 'Name': 'string',
 | 'Values': [
 | 'string',
 | ]
 | },
 | ]

import boto3
instances = ec2.instances.filter(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
for instance in instances:
    print(, instance.instance_type)