Download S3 files using PowerShell

Use the below script to download the files from any S3 bucket to your local machine

$sourceBucket = '<bucket name from where you want to copy the files from>'
$profile = '<aws profile name>'
$Folder = '<Folder Path on local drive>'

$items = Get-S3Object -BucketName $sourceBucket -ProfileName $profile -Region 'us-east-1'
Write-Host "$($items.Length) objects to copy"
$index = 1
$items | % {
    Write-Host "$index/$($items.Length): $($_.Key)"
    $fileName = $Folder + ".\$($_.Key.Replace('/','\'))"
    Write-Host "$fileName"
    Read-S3Object -BucketName $sourceBucket -Key $_.Key -File $fileName -ProfileName $profile -Region 'us-east-1' > $null
    $index += 1
}

Disclosure: We use affiliate links to monetize our content.  We may receive a commission on products or services that you purchase through clicking on links within this blog.

Find files of given size using PowerShell

Get-ChildItem

The cmdlet can be used to get the items and the child items in a  particular location, folder or a directory.

Get-ChildItem [[-Path] <String[]>]

In order to get the files in a folder or subfolder, you can use this cmdlet.

To traverse the sub folders, you can use -Recurse parameter and to limit the number of levels that needs to be recursed, you can use -Depth parameter.

To get the list of files greater than, less than or equal to, you can use Where-Object. It allows to select an object from a collection based on some condition.

Get-ChildItem [path] -recurse | where-object {$_.length -gt [size]} | Sort-Object length

Example

Get-ChildItem ‘C:\Users\ImgToUpload1 – Copy’ -recurse | where-object {$_.length -gt 2000000} | Sort-Object length

This will list all the files in ‘C:\Users\ImgToUpload1 – Copy’ folder that are greater than 2MB in size and also sort them based on the size or length in ascending order (default order). In order to sort the list in descending order, you can use -Descending.

Example

Get-ChildItem ‘C:\Users\ImgToUpload1 – Copy’ -recurse | where-object {$_.length -gt 2MB} | Sort-Object length -Descending

You can use

• -lt for less than
• -eq for equal to
• -ne for not equal to
• -ge for greater than or equal to
• -le for less than or equal to

Note: ‘=’ sign is not used to check equality in PowerShell as it is used as an assignment operator

PowerShell – Working with URLs

Invoke-WebRequest

This command is used to get the content from a web page by sending the HTTP or HTTPS requests to that page. In other words, you can simple parse or scrape a web page for images, links, etc.

In response, you will see a parsed result including status code, links, images, input fields, etc.

Invoke-WebRequest <Uri>

Example:

Invoke-WebRequest https://google.com

SSL/TLS

By default, PowerShell uses TLS 1.0 for HTTP / HTTPS requests.

You can easily check the TLS version of any website.

Google Chrome – Click F12 > Go to Security Tab

Mozilla Firefox – Click on the padlock icon on the left of address bar and then click on the right arrow as shown below.

Click on more information and you will see a popup with TLS version.

Internet Explorer – Open a website > Right click on page > Click on Properties

Not all the websites use TLS 1.0 and it is not possible to establish a secure connection to a website using different TLS version.

So in order to make HTTP or HTTPS calls using Invoke-WebRequest, you have to force PowerShell to use a different version (eg. TLS 1.2)

# Forcing PowerShell to use TLS 1.2

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Example:

If you are wondering, which versions you can use with PowerShell, then simply use the following cmdlet

[enum]::GetNames([Net.SecurityProtocolType])