Boto 3: Basic and setup

Boto : Boto is a SDK designed to improve the use of the python programming language in aws.

Setup requirement:

  • Aws signup
  • Python version 2.7
  • Pycharm 3.3 [ide]
  • Pip setup

For programmatic access: We need to enable the access in the aws iam:

For programmatically user access , secret key and access id.

Setup awscli on windows

  • Awscli
  • Prerequires:
    • Check your system has Python 2.7
    • Pip is configured

Configure awscli

https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html

Configure the access key and id

C:\Users\amitm>aws configure

AWS Access Key ID [****************Z2FA]:

AWS Secret Access Key [****************V270]:

Default region name [ap-south-1]:

Default output format [test]:

Check setup?

C:\Users\amitm>aws s3 ls

2018-10-15 13:08:20 cf-templates-106h68kzl5m34-us-east-2

2018-11-08 23:39:48 openwriteup

2018-11-08 23:46:12 openwriteup-1

2018-11-09 00:16:44 test-openwriteup

What is awscli??

  • This is a command line tool
  • If we are writing script we can use it
  • Testing purpose or want to use shell or powershell it is useful that

Setup boto3

  • Pip install boto3
  • Test boto3
    • Python
    • Import boto3
    • Help(boto3)

Botocore

  • A low-level interface to a growing number of Amazon Web Services. The botocore package is the foundation for the AWS CLIas well as boto3.
  • Botocore provides the low level clients, session, and credential & configuration data. Boto 3 builds on top of Botocore by providing its own session, resources and collections.
  • botocore does not provide higher-level abstractions on top of these services, operations and responses. That is left to the application layer. The goal of botocore is to handle all of the low-level details of making requests and getting results from a service

Core concepts of boto3

Resources

  • higher-level, object-oriented API
  • generated from resource description
  • uses identifiers and attributes
  • has actions (operations on resources)
  • exposes subresources and collections

example:

import boto3

s3 = boto3.resource('s3')

bucket = s3.Bucket('mybucket')

for obj in bucket.objects.all():

print(obj.key, obj.last_modified)

Boto Client:

  • low-level service access
  • generated from service description
  • exposes botocore client to the developer
  • typically maps 1:1 with the service API
  • snake-cased method names (e.g. ListBuckets API => list_buckets method)

example:

import boto3

client = boto3.client('s3')

response = client.list_objects(Bucket='mybucket')

for content in response['Contents']:

obj_dict = client.get_object(Bucket='mybucket', Key=content['Key'])

print(content['Key'], obj_dict['LastModified'])

 

Difference Between resource and client:

Resource object is very high level object, every operation with resource object would be high level operation. We may not have all the operation with resource.

Client is low level object, so whatever operation we want to perform its always be available. Client operations are mostly dictionary operation.

Session:

  • stores configuration information (primarily credentials and selected region)
  • allows you to create service clients and resources

Simple object to get it connected to particular aws account or iam account. If i want to connect any iam acocunt, session object will be used.

Pagination

  • Automatically handles pagination
  • Yields individual pages
  • You must process each pages

Example: I have three thousand object in my s3 bucket, which i want to list. Boto3 Api can only list till a limit (1000 object). In such cases paginator can be used to list all the 3k objects. It will be using 3 pages to list .

Waiter

Waiter are used for reach waiting to reach certain state

Example: I have ec2 instance, which i newly launched, it takes some time to reach running state. For that purpose we can use waiter

 

VMware VCSA 6.x backup using ansible

Recently, I have one project requirement where i have to use vmware rest api and ansible to take the vcsa backup. In this blog we will first explore vcsa apiexplorer and traverse to rest api for backup

we need to open: https:<vcsa fqdn or ip>/apiexplorer

Select the appliance from the drop down and traverse to the backup job

explore POST /appliance/recover/backup/job . It will have parameter values

These are the option we will be using while coding in ansible. parts option you can get from vcsa vami console.

SEAT is optional.

After exploring api, its time to look into ansible. First part would be login to vcsa apiexplore. This require authentication.

VCSA login using rest api /ansible code snip

As explain above for backup request body, same way for ansible we need write json file

Now we need to write the backup task in ansible yaml file

So backup.yaml will contain login and backup task. Backup task will be calling json file , which contain vcsa backup config info.

HPONCFG : To delete local user and add AD user

New requirement came for ilo3 and il04 for hpservers, below are the requirements:

-Remove local user

-Add Ad user.

We can add it manually, but the requirement for huge setup of server. After searching, I found hponcfg tool, which runs through xml. Below is the reference link, This links has all the xml files

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c03219637

Delete local user xml. We need to run this xml with hpocnfg tool (which is available for windows version as well)

 

 

 

 

 

Check VMware VCSA certs validity using Ansible and RestAPI

Recently a requirement came, where VMware VCSA 6.x compliance need to be checked using Ansible.  Vendor wants to use VMware RestApi [Not interested to use VMware Python SDK].

Note: VMware Ansible module comes with VMware Python SDK [PyVmomi]. This compliance check was having

ssh , ipv6, ntp, CA certs, DNS check and set (in case of value has changed).  Most of the option was available except CA cert check. Below yaml code has three parts;

  • VCSA login
  • cert fetch
  • logging

—-vc_cert_check.yaml–

This is used for Product hardening!!


 

pyVmomi module: Script for fetching hardware information from ESXi

import argparse
from pyVmomi import vim
from pyVim.connect import SmartConnect,Disconnect
import atexit
import ssl

def validate_options():
  parser = argparse.ArgumentParser(description='input parameters')
  parser.add_argument('-d','--dest_host',dest='dhost',required=True,help='The ESxi destination host IP')
  parser.add_argument('-v','--vc_host',dest='vchost',required=False,help='The VC ip')
  parser.add_argument('-u','--vc_user',dest='vcuser',required=True,help='VC username')
  parser.add_argument('-p','--vc_pass',dest='vcpasswd',required=True,help='VC passwd')
  args = parser.parse_args()
  return args

def getHostID(content,dhost):
  if content.searchIndex.FindByIp(None,dhost,False):
    host = content.searchIndex.FindByIp(None,dhost,False)
  else:
    host = content.searchIndex.FindByDnsName(None,dhost,False)
  return host

def get_HostInfo(content,dhost):
   search_index = content.searchIndex
   root_folder =  content.rootFolder
   view_ref = content.viewManager.CreateContainerView(container=root_folder,type=[vim.HostSystem], recursive=True)
   host = view_ref.view[0]
   #print host.name
   print 'UUID INFO %s' %(host.summary.hardware.uuid)
   print 'Hardware Model %s' %(host.summary.hardware.model)
   print '%s Server has %s Biosversion'%(host.hardware.biosInfo.vendor,host.hardware.biosInfo.biosVersion)
   pcilist=host.hardware.pciDevice
   print '{0}'.format("Vendor Name").ljust(20)+ '{0}'.format("Device Name").ljust(120)+ '{0}'.format("Slot").ljust(30)+ '{0}'.format('Device ID').ljust(10)
   print '*************************************************************************************************************************************************************************************************'
   for i in pcilist:
     a = i.vendorName
     b = i.deviceName
     c = i.deviceId
     d = i.slot
     print '{0}'.format(a).ljust(20)+ '{0}'.format(b).ljust(120)+ '{0}'.format(d).ljust(30)+ '{0}'.format(c).ljust(10)
     #print '%s has devicename %s and  device ID %s'%(i.vendorName,i.deviceName,i.deviceId)
   print '*********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************' 
def main():
  opts = validate_options()
  if opts.vchost:
    print 'Connecting to vcenter and collecting sensor info for %s' %opts.dhost
  else:
    print 'Connecting to esxi host for %s' %opts.dhost
    opts.vchost = opts.dhost
  si = SmartConnect(host=opts.vchost, user=opts.vcuser, pwd=opts.vcpasswd)
  content = si.RetrieveContent()
  #print content
  hostinfo = get_HostInfo(content,opts.dhost)
  hostid = getHostID(content,opts.dhost)
  sensorinfo=hostid.runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo
  print '{0}'.format("Sensor").ljust(30)+ '{0}'.format("Sensor Detail").ljust(90)+ '{0}'.format('Status').ljust(10)+ '{0}'.format('Reading').ljust(10) +'{0}'.format('Units').ljust(13)+ '{0}'.format('Summary').ljust(20)
  print '**************************************************************************************************************************************************************************************************************'
  for i in sensorinfo:
    j = i.healthState
    a=str(i.currentReading)
    b=i.baseUnits
    c=i.sensorType
    print '{0}'.format(c).ljust(30)+ '{0}'.format(i.name).ljust(90)+ '{0}'.format(j.label).ljust(10)+ '{0}'.format(a).ljust(10) + '{0}'.format(b).ljust(13)+ '{0}'.format(j.summary).ljust(20)

if __name__ =='__main__':
  main()
How to run this script : 

python <name of script> -v <vc server> -d <esxi host which hardware want to list> -u <vc user name> -p <vc password>

This script is written in python. I have used the  pyVmomi module.

pyVmomi rpm for centos7

pyVmomi is the Python SDK for the VMware vSphere API that allows you to manage ESX, ESXi, and vCenter. pyVmomi is available on git.

https://github.com/vmware/pyvmomi

I have created a rpm format of same pyVmomi SDK for centos7.  This rpm will be installed in the /opt folder on your centos 7.

Below is the Spec file:

%define BUILD pyvmomi_master.1.0.1.x86_64
Summary: Pyvmomi package
Name: pyvmomi_master
Release: 1.0
Version: 1
License: Apache License 2.0
Requires: python-six
Requires: python-requests
Requires: python-setuptools
BuildArch: noarch

%description
This package contains the vSphere python SDK

%post
%files
%defattr(-,root,root,-)
/opt/pyvmomi-master
%doc
%changeLog
* Fri Jul 14 2017 Amit <amit@openwriteup.com> 1-1.0
- Pyvmomi 6.5

Once you install the rpm, it will be in the /opt/pyvmomi-master folder.

 rpm -ivh pyvmomi_master-1-1.0.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:pyvmomi_master-1-1.0             ################################# [100%]


################################# [100%]
[root@devbox noarch]# ls /opt/pyvmomi-master/
docs  LICENSE.txt  MANIFEST.in  NOTICE.txt  pyVim  pyVmomi  README.rst  requirements.txt  sample  setup.cfg  setup.py  test-requirements.txt  tests  tox.ini

Post installation of the package, we need to run following step:

[root@devbox pyvmomi-master]# python setup.py –help
Common commands: (see ‘–help-commands’ for more)

setup.py build      will build the package underneath ‘build/’
setup.py install    will install the package

 python setup.py install
running install
running bdist_egg
running egg_info
creating pyvmomi.egg-info
writing requirements to pyvmomi.egg-info/requires.txt
writing pyvmomi.egg-info/PKG-INFO
writing top-level names to pyvmomi.egg-info/top_level.txt
writing dependency_links to pyvmomi.egg-info/dependency_links.txt
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest file 'pyvmomi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyvmomi.egg-info/SOURCES.txt'

vSphere SDK for python : pyVmomi

VMware has released pyVmomi for developers who wants to code in python. pyVmomi is a VMware SDK to manage ESXi and vCenter using python.
Using pip we can install:
pip install pyvmomi

Developers can download the pyvmomi package from python.org

https://pypi.python.org/pypi/pyvmomi

When we extract and list the package
LICENSE.txt NOTICE.txt pyVim pyvmomi.egg-info requirements.txt setup.cfg test-requirements.txt tox.ini MANIFEST.in PKG-INFO pyVmomi README.rst sample setup.py tests

Perform the below steps:
 python setup.py install
 python setup.py develop

pyVmomi package provide mainly two modules : pyVim and pyVmomi
Let’s explore pyVim. This module will be used to connect to ESXi and vSphere Center.
>>> import pyVim
>>> help(pyVim)
PACKAGE CONTENTS
connect
task
>>>import pyVim.connect
NAME
pyVim.connect – Connect to a VMOMI ServiceInstance.
>>>help(pyVim.connect)
This contents two function to connect service instance: SmartConnect and Connect
SmartConnect(protocol=’https’, host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, path=’/sdk’, preferredApiVersions=None, keyFile=None, certFile=N
one, thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Determine the most preferred API version supported by the specified server,then connect to the specified server using that API version, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access.Clients should modify the service parameter only when connecting to a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine.

Connect(host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, adapter=’SOAP’, namespace=None, path=’/sdk’, version=None, keyFile=None, certFile=None,
thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Connect to the specified server, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access. Clients should modify the service parameter only when connecting to
a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine

 

We can connect using SmartConnect
From pyVim.connect import SmartConnect
conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn

'vim.ServiceInstance:ServiceInstance'

This is service instance type of managed object. For writing code further we can make use of vSphere mob interface: http://<vc-ip>/mob

mob-2

Wrote a small code which fetch the build number and name of datecenter

#!/bin/python
from pyVim.connect import SmartConnect
conn = SmartConnect(host="<VC hostname>",user="<username>",pwd="<password>")
print ("Multi host is supported")
print (conn.capability.multiHostSupported)
print ("----------------------------------------------------------")
print ("VC System Current time")
print (conn.CurrentTime())
print ("----------------------------------------------------------")
print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (conn.content.about.build)
print ("----------------------------------------------------------")
datacenter = conn.content.rootFolder.childEntity
print ("Name of datacenters")
print ("----------------------------------------------------------")
for i in datacenter:
 print (i.name)
# python vcconnect.py 
Multi host is supported
True
----------------------------------------------------------
VC System Current time
2016-09-22 14:33:17.
----------------------------------------------------------
VC server build info
6.0.0
VMware vCenter Server 6.0.0 build-xxxxxx
----------------------------------------------------------
Name of datacenters
----------------------------------------------------------
test_Lab

Below are the screenshot how I traverse for Version number:

conn.content.about.version

conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>)

Print conn

'vim.ServiceInstance:ServiceInstance'

conn is Service Instance Object. listing the properties and methods from mob interface.

mob-3

In properties segment, I am listing the content property.This content the about property.

mob-4

In about we can list version:

mob-5

That’s the way below code has traversed.

print ("VC server build info")
print (conn.content.about.version)
print (conn.content.about.fullName)
print (conn.content.about.build)

 

List all the running instance on amazon VPC

In my test environment, I have amazon VPC, which I am accessing using Linux server.

For performing all the activities on amazon vpc, I have used python script.For automation in amazon VPC, aws provides module boto3, which need to be installed using python pip. Using this module we list all the running instances.In below script I am creating a config file and then reading that config file.Following script perform all these steps:

  • Create a config file, which is required to connect aws vpc.
  • Read the config file, and list all the instance following details:
    • ‘Name’:name,
    • ‘Type’: instance.instance_type,
    • ‘State’:instance.state[‘Name’],
    • ‘Private IP’:instance.private_ip_address,
    • ‘Public IP’: instance.public_ip_address,
    • ‘Launch Time’: instance.launch_time

 

</script>

import ConfigParser,boto3,os,sys,paramiko
from collections import defaultdict
config = ConfigParser.RawConfigParser()
#When adding sections or items, add them in reverse order
config.add_section('EC2')
config.add_section('USER')
config.set('EC2','SSHKey','<keyname>')
config.set('EC2','VPC_IP','<vpcname>')
config.set('EC2','Security_Group','<securitygroupname>')
config.set('EC2','DisableAPI_Termination','False')
config.set('USER','Username','<ec2user>')
config.set('USER','AWS_Profile','<aws user to login>')
config.set('EC2','Region','<aws region>')
config.set('USER','Private_Key','<path_to_privatekey>')

#Writing configruation to config file
name = raw_input("Enter the config file name::: ")
with open(name, 'wb') as configfile:
config.write(configfile)
#Reading the config file
config1 = ConfigParser.ConfigParser()
config1.read(name)
ses = boto3.Session(profile_name = config1.get("USER", "AWS_Profile"))
ec2 = ses.resource('ec2')
key = paramiko.rsakey.RSAKey.from_private_key_file(filename=config1.get("USER","Private_Key")) 
running_instances = ec2.instances.filter(Filters=[{
'Name': 'instance-state-name',
'Values': ['running']}])
ec2info = defaultdict()
for instance in running_instances:
for tag in instance.tags:
if 'Name' in tag['Key']:
name = tag['Value']
ec2info[instance.id] = {
'Name':name,
'Type': instance.instance_type,
'State':instance.state['Name'],
'Private IP':instance.private_ip_address,
'Public IP': instance.public_ip_address,
'Launch Time': instance.launch_time
}
attributes = ['Name','Type','State','Private IP','Public IP','Launch Time']
for instance_id, instance in ec2info.items():
for key in attributes:
print("{0}:{1}".format(key,instance[key]))
print("------")

 

output of the script :

——
Name:testinstance
Type:m4.xlarge
State:running
Private IP:10.140.30.209
Public IP:None
Launch Time:2016-08-26 23:09:17+00:00
——

 

 

aws instance listing using python sdk

This blog is for those, who are very new to aws and python. They want to start both of them together. Assuming they have setup boto3 environment in their test lab.

In lab setup type python: python

It will give python prompt, we can explore boto3.

>>> import boto3
>>> dir(boto3)
[‘DEFAULT_SESSION’, ‘NullHandler’, ‘Session’, ‘__author__’, ‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘__version__’, ‘_get_default_session’, ‘client’, ‘docs’, ‘exceptions’, ‘logging’, ‘resource’, ‘resources’, ‘session’, ‘set_stream_logger’, ‘setup_default_session’, ‘utils’]

Perform help (boto3) ,It will show the package content with this package..

PACKAGE CONTENTS
compat
docs (package)
dynamodb (package)
ec2 (package)
exceptions
resources (package)
s3 (package)
session
utils

Lets import the resources : from boto3 import resources

>>> dir (boto3.resources)
[‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘action’, ‘base’, ‘collection’, ‘factory’, ‘model’, ‘params’, ‘response’]
>>>

PACKAGE CONTENTS
action
base
collection
factory
model
params
response

 

/*perform following on your python console or write a .py script
import boto3
ec2=boto3.resource(ec2)
#help(ec2) /*it will list all the available option with ec2*/
#help(ec2.instances) /*search for filter*/
#help(ec2.instances.filter /*list the filter option and list syntax
/* instance_iterator = ec2.instances.filter(
 | DryRun=True|False,
 | InstanceIds=[
 | 'string',
 | ],
 | Filters=[
 | {
 | 'Name': 'string',
 | 'Values': [
 | 'string',
 | ]
 | },
 | ]
*/


import boto3
ec2=boto3.resource('ec2')
instances = ec2.instances.filter(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
for instance in instances:
    print(instance.id, instance.instance_type)