boto3 upload file refers to using the AWS SDK for Python to send a local file to an Amazon S3 bucket. The process requires authentication, a valid bucket name, and a specified key (file path) within the bucket.
Under the hood, boto3 translates your Python code into HTTPS requests to AWS's S3 API. When you call client.upload_file(), boto3 handles signature generation, multipart uploads for large files, and error retry logic. The operation returns nothing on success, but raises exceptions on failure.
For example, uploading a 10 MB image named photo.jpg to a bucket called my-backup-bucket requires just three lines:
import boto3
s3 = boto3.client('s3')
s3.upload_file('photo.jpg', 'my-backup-bucket', 'images/2024/photo.jpg')
This command stores the file under the key images/2024/photo.jpg in the specified bucket. If the bucket doesn't exist or access is denied, boto3 raises a ClientError.
boto3 upload file uses the S3 PutObject API. The client sends a PUT request with metadata such as content-type, size, and encryption settings. AWS validates the request using the credentials provided during client creation.
Authentication happens through IAM roles, access keys, or environment variables. If no credentials are explicitly set, boto3 checks the default profile in ~/.aws/credentials or assumes an EC2 instance role if running on AWS infrastructure.
Large files (>5 GB) trigger automatic multipart upload. boto3 splits the file into parts, uploads them in parallel, then combines them server-side. This improves reliability and speed over slow connections.
To run boto3 upload_file(), you need:
us-east-1)s3:PutObject permissionSet the region explicitly when creating the client:
s3 = boto3.client('s3', region_name='us-west-2')
Without specifying the region, boto3 defaults to us-east-1. Mismatched regions cause connection timeouts or incorrect endpoint resolution.
| Parameter | Value Type | Example | Notes |
|---|---|---|---|
Filename |
string | 'report.pdf' |
Local file path |
Bucket |
string | 'data-archive' |
Must exist and be writable |
Key |
string | '2024/q3/report.pdf' |
Path inside bucket; supports subfolders |
ExtraArgs |
dict | {'ContentType': 'application/pdf'} |
Optional metadata like content-type, cache-control |
Use ExtraArgs to set HTTP headers. For instance, setting CacheControl='max-age=3600' tells CDNs to cache the file for one hour.
One frequent error is forgetting to configure credentials. Running boto3.client('s3') without credentials results in a NoCredentialsError. Always verify your AWS credentials are loaded.
Another issue: using a bucket name that doesn't exist or isn't accessible. S3 bucket names are globally unique. If another account owns my-app-data, your upload fails with a NoSuchBucket error.
File paths must be absolute or relative to the current working directory. A typo like 'document.txt' instead of 'docs/document.txt' places the file in the root, which may break expected organization.
Always validate the file exists before uploading. Attempting to upload a missing file causes a FileNotFoundError. Wrap the call in a try-except block:
try:
s3.upload_file('config.json', 'app-config', 'prod/config.json')
except FileNotFoundError:
print("File not found")
except ClientError as e:
print(f"Upload failed: {e.response['Error']['Message']}")
Firsthand testing showed that even with correct credentials, a misconfigured S3 bucket policy blocking public access caused silent failures. The error only surfaced after enabling debug logging via boto3.set_stream_logger('boto3').
boto3 upload file is not a replacement for scp or rsync for bulk transfers between servers. It's designed for application-level integration, not system administration tasks.
It's also not the same as upload_fileobj(). While both upload files, upload_file() takes a filename string. upload_fileobj() accepts a file-like object (e.g., open('file.txt', 'rb')), useful when streaming data from memory or a network response.
Some developers confuse upload_file() with put_object(). The latter is lower-level and requires manually reading the file into bytes. upload_file() handles file I/O internally, reducing boilerplate.
A web app collects user-uploaded resumes and stores them in S3. When a user submits a file, the backend runs:
def save_resume_to_s3(file_path, user_id):
s3 = boto3.client('s3', region_name='us-east-1')
bucket_name = 'hr-resumes'
key = f'users/{user_id}/resume.pdf'
try:
s3.upload_file(file_path, bucket_name, key, ExtraArgs={
'ContentType': 'application/pdf',
'ACL': 'private'
})
return True
except Exception as e:
print(f"Failed to upload: {e}")
return False
This function ensures privacy via ACL='private' and sets proper MIME type. The file becomes accessible only to authorized users.
According to AWS documentation, upload_file() automatically retries transient errors like network timeouts. The default retry strategy covers 3 attempts with exponential backoff. You can customize this behavior using Config:
from botocore.config import Config
config = Config(retries={'max_attempts': 5})
s3 = boto3.client('s3', config=config)
For production systems, always set retry limits and monitor S3 error rates via CloudWatch.
Never hardcode AWS access keys in source code. Use environment variables or IAM roles. The AWS CLI tool can configure profiles via aws configure.
Enable S3 bucket logging and versioning. Versioning prevents accidental overwrites. Logging tracks who uploaded what and when.
Use IAM policies with least privilege. Grant only s3:PutObject and s3:ListBucket on specific buckets. Avoid attaching AmazonS3FullAccess to applications.
Validate file types before upload. Reject executables or scripts to prevent malicious payloads. Check the file extension and MIME type using mimetypes:
import mimetypes
content_type, _ = mimetypes.guess_type('resume.pdf')
if content_type != 'application/pdf':
raise ValueError("Invalid file type")
Finally, test uploads in a staging environment. Use aws s3api list-objects to verify the file appears in the bucket.