Setting up pyCharm
- Install pycharm
https://www.jetbrains.com/pycharm/
Go to pycharm configure->setting
Default project->Python interpreter
Add python interpreter
install boto3 package
Now create new package
Setting up pyCharm
https://www.jetbrains.com/pycharm/
Go to pycharm configure->setting
Default project->Python interpreter
Add python interpreter
install boto3 package
Now create new package
Boto : Boto is a SDK designed to improve the use of the python programming language in aws.
Setup requirement:
For programmatic access: We need to enable the access in the aws iam:
For programmatically user access , secret key and access id.
Setup awscli on windows
Configure awscli
https://docs.aws.amazon.com/cli/latest/userguide/install-windows.html
Configure the access key and id
C:\Users\amitm>aws configure
AWS Access Key ID [****************Z2FA]: AWS Secret Access Key [****************V270]: Default region name [ap-south-1]: Default output format [test]:
Check setup?
C:\Users\amitm>aws s3 ls
2018-10-15 13:08:20 cf-templates-106h68kzl5m34-us-east-2 2018-11-08 23:39:48 openwriteup 2018-11-08 23:46:12 openwriteup-1 2018-11-09 00:16:44 test-openwriteup
What is awscli??
Setup boto3
Botocore
Core concepts of boto3
Resources
example:
import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('mybucket') for obj in bucket.objects.all(): print(obj.key, obj.last_modified)
Boto Client:
example:
import boto3 client = boto3.client('s3') response = client.list_objects(Bucket='mybucket') for content in response['Contents']: obj_dict = client.get_object(Bucket='mybucket', Key=content['Key']) print(content['Key'], obj_dict['LastModified'])
Difference Between resource and client:
Resource object is very high level object, every operation with resource object would be high level operation. We may not have all the operation with resource.
Client is low level object, so whatever operation we want to perform its always be available. Client operations are mostly dictionary operation.
Session:
Simple object to get it connected to particular aws account or iam account. If i want to connect any iam acocunt, session object will be used.
Pagination
Example: I have three thousand object in my s3 bucket, which i want to list. Boto3 Api can only list till a limit (1000 object). In such cases paginator can be used to list all the 3k objects. It will be using 3 pages to list .
Waiter
Waiter are used for reach waiting to reach certain state
Example: I have ec2 instance, which i newly launched, it takes some time to reach running state. For that purpose we can use waiter
Recently, I have one project requirement where i have to use vmware rest api and ansible to take the vcsa backup. In this blog we will first explore vcsa apiexplorer and traverse to rest api for backup
we need to open: https:<vcsa fqdn or ip>/apiexplorer
Select the appliance from the drop down and traverse to the backup job
explore POST /appliance/recover/backup/job . It will have parameter values
These are the option we will be using while coding in ansible. parts option you can get from vcsa vami console.
After exploring api, its time to look into ansible. First part would be login to vcsa apiexplore. This require authentication.
As explain above for backup request body, same way for ansible we need write json file
Now we need to write the backup task in ansible yaml file
So backup.yaml will contain login and backup task. Backup task will be calling json file , which contain vcsa backup config info.
New requirement came for ilo3 and il04 for hpservers, below are the requirements:
-Remove local user
-Add Ad user.
We can add it manually, but the requirement for huge setup of server. After searching, I found hponcfg tool, which runs through xml. Below is the reference link, This links has all the xml files
https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c03219637
Delete local user xml. We need to run this xml with hpocnfg tool (which is available for windows version as well)
Recently a requirement came, where VMware VCSA 6.x compliance need to be checked using Ansible. Vendor wants to use VMware RestApi [Not interested to use VMware Python SDK].
Note: VMware Ansible module comes with VMware Python SDK [PyVmomi]. This compliance check was having
ssh , ipv6, ntp, CA certs, DNS check and set (in case of value has changed). Most of the option was available except CA cert check. Below yaml code has three parts;
—-vc_cert_check.yaml–
This is used for Product hardening!!
import argparse
from pyVmomi import vim
from pyVim.connect import SmartConnect,Disconnect
import atexit
import ssl
def validate_options():
parser = argparse.ArgumentParser(description='input parameters')
parser.add_argument('-d','--dest_host',dest='dhost',required=True,help='The ESxi destination host IP')
parser.add_argument('-v','--vc_host',dest='vchost',required=False,help='The VC ip')
parser.add_argument('-u','--vc_user',dest='vcuser',required=True,help='VC username')
parser.add_argument('-p','--vc_pass',dest='vcpasswd',required=True,help='VC passwd')
args = parser.parse_args()
return args
def getHostID(content,dhost):
if content.searchIndex.FindByIp(None,dhost,False):
host = content.searchIndex.FindByIp(None,dhost,False)
else:
host = content.searchIndex.FindByDnsName(None,dhost,False)
return host
def get_HostInfo(content,dhost):
search_index = content.searchIndex
root_folder = content.rootFolder
view_ref = content.viewManager.CreateContainerView(container=root_folder,type=[vim.HostSystem], recursive=True)
host = view_ref.view[0]
#print host.name
print 'UUID INFO %s' %(host.summary.hardware.uuid)
print 'Hardware Model %s' %(host.summary.hardware.model)
print '%s Server has %s Biosversion'%(host.hardware.biosInfo.vendor,host.hardware.biosInfo.biosVersion)
pcilist=host.hardware.pciDevice
print '{0}'.format("Vendor Name").ljust(20)+ '{0}'.format("Device Name").ljust(120)+ '{0}'.format("Slot").ljust(30)+ '{0}'.format('Device ID').ljust(10)
print '*************************************************************************************************************************************************************************************************'
for i in pcilist:
a = i.vendorName
b = i.deviceName
c = i.deviceId
d = i.slot
print '{0}'.format(a).ljust(20)+ '{0}'.format(b).ljust(120)+ '{0}'.format(d).ljust(30)+ '{0}'.format(c).ljust(10)
#print '%s has devicename %s and device ID %s'%(i.vendorName,i.deviceName,i.deviceId)
print '*********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************'
def main():
opts = validate_options()
if opts.vchost:
print 'Connecting to vcenter and collecting sensor info for %s' %opts.dhost
else:
print 'Connecting to esxi host for %s' %opts.dhost
opts.vchost = opts.dhost
si = SmartConnect(host=opts.vchost, user=opts.vcuser, pwd=opts.vcpasswd)
content = si.RetrieveContent()
#print content
hostinfo = get_HostInfo(content,opts.dhost)
hostid = getHostID(content,opts.dhost)
sensorinfo=hostid.runtime.healthSystemRuntime.systemHealthInfo.numericSensorInfo
print '{0}'.format("Sensor").ljust(30)+ '{0}'.format("Sensor Detail").ljust(90)+ '{0}'.format('Status').ljust(10)+ '{0}'.format('Reading').ljust(10) +'{0}'.format('Units').ljust(13)+ '{0}'.format('Summary').ljust(20)
print '**************************************************************************************************************************************************************************************************************'
for i in sensorinfo:
j = i.healthState
a=str(i.currentReading)
b=i.baseUnits
c=i.sensorType
print '{0}'.format(c).ljust(30)+ '{0}'.format(i.name).ljust(90)+ '{0}'.format(j.label).ljust(10)+ '{0}'.format(a).ljust(10) + '{0}'.format(b).ljust(13)+ '{0}'.format(j.summary).ljust(20)
if __name__ =='__main__':
main()
How to run this script : python <name of script> -v <vc server> -d <esxi host which hardware want to list> -u <vc user name> -p <vc password>
This script is written in python. I have used the pyVmomi module.
pyVmomi is the Python SDK for the VMware vSphere API that allows you to manage ESX, ESXi, and vCenter. pyVmomi is available on git.
https://github.com/vmware/pyvmomi
I have created a rpm format of same pyVmomi SDK for centos7. This rpm will be installed in the /opt folder on your centos 7.
Below is the Spec file:
%define BUILD pyvmomi_master.1.0.1.x86_64 Summary: Pyvmomi package Name: pyvmomi_master Release: 1.0 Version: 1 License: Apache License 2.0 Requires: python-six Requires: python-requests Requires: python-setuptools BuildArch: noarch %description This package contains the vSphere python SDK %post %files %defattr(-,root,root,-) /opt/pyvmomi-master %doc %changeLog * Fri Jul 14 2017 Amit <amit@openwriteup.com> 1-1.0 - Pyvmomi 6.5
Once you install the rpm, it will be in the /opt/pyvmomi-master folder.
rpm -ivh pyvmomi_master-1-1.0.noarch.rpm Preparing... ################################# [100%] Updating / installing... 1:pyvmomi_master-1-1.0 ################################# [100%] ################################# [100%]
[root@devbox noarch]# ls /opt/pyvmomi-master/ docs LICENSE.txt MANIFEST.in NOTICE.txt pyVim pyVmomi README.rst requirements.txt sample setup.cfg setup.py test-requirements.txt tests tox.ini
Post installation of the package, we need to run following step:
[root@devbox pyvmomi-master]# python setup.py –help
Common commands: (see ‘–help-commands’ for more)
setup.py build will build the package underneath ‘build/’
setup.py install will install the package
python setup.py install running install running bdist_egg running egg_info creating pyvmomi.egg-info writing requirements to pyvmomi.egg-info/requires.txt writing pyvmomi.egg-info/PKG-INFO writing top-level names to pyvmomi.egg-info/top_level.txt writing dependency_links to pyvmomi.egg-info/dependency_links.txt writing manifest file 'pyvmomi.egg-info/SOURCES.txt' reading manifest file 'pyvmomi.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'pyvmomi.egg-info/SOURCES.txt'
VMware has released pyVmomi for developers who wants to code in python. pyVmomi is a VMware SDK to manage ESXi and vCenter using python.
Using pip we can install:
pip install pyvmomi
Developers can download the pyvmomi package from python.org
https://pypi.python.org/pypi/pyvmomi
When we extract and list the package
LICENSE.txt NOTICE.txt pyVim pyvmomi.egg-info requirements.txt setup.cfg test-requirements.txt tox.ini MANIFEST.in PKG-INFO pyVmomi README.rst sample setup.py tests
Perform the below steps: python setup.py install python setup.py develop
pyVmomi package provide mainly two modules : pyVim and pyVmomi
Let’s explore pyVim. This module will be used to connect to ESXi and vSphere Center.
>>> import pyVim
>>> help(pyVim)
PACKAGE CONTENTS
connect
task
>>>import pyVim.connect
NAME
pyVim.connect – Connect to a VMOMI ServiceInstance.
>>>help(pyVim.connect)
This contents two function to connect service instance: SmartConnect and Connect
SmartConnect(protocol=’https’, host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, path=’/sdk’, preferredApiVersions=None, keyFile=None, certFile=N
one, thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Determine the most preferred API version supported by the specified server,then connect to the specified server using that API version, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access.Clients should modify the service parameter only when connecting to a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine.
Connect(host=’localhost’, port=443, user=’root’, pwd=”, service=’hostd’, adapter=’SOAP’, namespace=None, path=’/sdk’, version=None, keyFile=None, certFile=None,
thumbprint=None, sslContext=None, b64token=None, mechanism=’userpass’)
Connect to the specified server, login and return the service instance object. Throws any exception back to caller. The service instance object is also saved in the library for easy access. Clients should modify the service parameter only when connecting to
a VMOMI server other than hostd/vpxd. For both of the latter, the default value is fine
We can connect using SmartConnect From pyVim.connect import SmartConnect conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>) Print conn 'vim.ServiceInstance:ServiceInstance'
This is service instance type of managed object. For writing code further we can make use of vSphere mob interface: http://<vc-ip>/mob
Wrote a small code which fetch the build number and name of datecenter
#!/bin/python from pyVim.connect import SmartConnect conn = SmartConnect(host="<VC hostname>",user="<username>",pwd="<password>") print ("Multi host is supported") print (conn.capability.multiHostSupported) print ("----------------------------------------------------------") print ("VC System Current time") print (conn.CurrentTime()) print ("----------------------------------------------------------") print ("VC server build info") print (conn.content.about.version) print (conn.content.about.fullName) print (conn.content.about.build) print ("----------------------------------------------------------") datacenter = conn.content.rootFolder.childEntity print ("Name of datacenters") print ("----------------------------------------------------------") for i in datacenter: print (i.name)
# python vcconnect.py Multi host is supported True ---------------------------------------------------------- VC System Current time 2016-09-22 14:33:17. ---------------------------------------------------------- VC server build info 6.0.0 VMware vCenter Server 6.0.0 build-xxxxxx ---------------------------------------------------------- Name of datacenters ---------------------------------------------------------- test_Lab
Below are the screenshot how I traverse for Version number:
conn.content.about.version
conn = SmartConnect(host=<”ESXi/vSphere Center ip”>, user = <”username”>, pwd= <”password”>) Print conn 'vim.ServiceInstance:ServiceInstance'
conn is Service Instance Object. listing the properties and methods from mob interface.
In properties segment, I am listing the content property.This content the about property.
In about we can list version:
That’s the way below code has traversed.
print ("VC server build info") print (conn.content.about.version) print (conn.content.about.fullName) print (conn.content.about.build)
In my test environment, I have amazon VPC, which I am accessing using Linux server.
For performing all the activities on amazon vpc, I have used python script.For automation in amazon VPC, aws provides module boto3, which need to be installed using python pip. Using this module we list all the running instances.In below script I am creating a config file and then reading that config file.Following script perform all these steps:
</script>
import ConfigParser,boto3,os,sys,paramiko from collections import defaultdict config = ConfigParser.RawConfigParser() #When adding sections or items, add them in reverse order config.add_section('EC2') config.add_section('USER') config.set('EC2','SSHKey','<keyname>') config.set('EC2','VPC_IP','<vpcname>') config.set('EC2','Security_Group','<securitygroupname>') config.set('EC2','DisableAPI_Termination','False') config.set('USER','Username','<ec2user>') config.set('USER','AWS_Profile','<aws user to login>') config.set('EC2','Region','<aws region>') config.set('USER','Private_Key','<path_to_privatekey>') #Writing configruation to config file name = raw_input("Enter the config file name::: ") with open(name, 'wb') as configfile: config.write(configfile) #Reading the config file config1 = ConfigParser.ConfigParser() config1.read(name) ses = boto3.Session(profile_name = config1.get("USER", "AWS_Profile")) ec2 = ses.resource('ec2') key = paramiko.rsakey.RSAKey.from_private_key_file(filename=config1.get("USER","Private_Key")) running_instances = ec2.instances.filter(Filters=[{ 'Name': 'instance-state-name', 'Values': ['running']}]) ec2info = defaultdict() for instance in running_instances: for tag in instance.tags: if 'Name' in tag['Key']: name = tag['Value'] ec2info[instance.id] = { 'Name':name, 'Type': instance.instance_type, 'State':instance.state['Name'], 'Private IP':instance.private_ip_address, 'Public IP': instance.public_ip_address, 'Launch Time': instance.launch_time } attributes = ['Name','Type','State','Private IP','Public IP','Launch Time'] for instance_id, instance in ec2info.items(): for key in attributes: print("{0}:{1}".format(key,instance[key])) print("------")
output of the script :
——
Name:testinstance
Type:m4.xlarge
State:running
Private IP:10.140.30.209
Public IP:None
Launch Time:2016-08-26 23:09:17+00:00
——
This blog is for those, who are very new to aws and python. They want to start both of them together. Assuming they have setup boto3 environment in their test lab.
In lab setup type python: python
It will give python prompt, we can explore boto3.
>>> import boto3
>>> dir(boto3)
[‘DEFAULT_SESSION’, ‘NullHandler’, ‘Session’, ‘__author__’, ‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘__version__’, ‘_get_default_session’, ‘client’, ‘docs’, ‘exceptions’, ‘logging’, ‘resource’, ‘resources’, ‘session’, ‘set_stream_logger’, ‘setup_default_session’, ‘utils’]
Perform help (boto3) ,It will show the package content with this package..
PACKAGE CONTENTS
compat
docs (package)
dynamodb (package)
ec2 (package)
exceptions
resources (package)
s3 (package)
session
utils
Lets import the resources : from boto3 import resources
>>> dir (boto3.resources)
[‘__builtins__’, ‘__doc__’, ‘__file__’, ‘__name__’, ‘__package__’, ‘__path__’, ‘action’, ‘base’, ‘collection’, ‘factory’, ‘model’, ‘params’, ‘response’]
>>>
PACKAGE CONTENTS
action
base
collection
factory
model
params
response
/*perform following on your python console or write a .py script import boto3 ec2=boto3.resource(ec2) #help(ec2) /*it will list all the available option with ec2*/ #help(ec2.instances) /*search for filter*/ #help(ec2.instances.filter /*list the filter option and list syntax /* instance_iterator = ec2.instances.filter( | DryRun=True|False, | InstanceIds=[ | 'string', | ], | Filters=[ | { | 'Name': 'string', | 'Values': [ | 'string', | ] | }, | ] */
import boto3 ec2=boto3.resource('ec2') instances = ec2.instances.filter( Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]) for instance in instances: print(instance.id, instance.instance_type)