Skip to content

Remove unnecessary getObjectMetadata() from getS3ObjectAndMetadata() #1060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
busbey merged 2 commits into from
Dec 2, 2017
Merged

Remove unnecessary getObjectMetadata() from getS3ObjectAndMetadata() #1060

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8cae420
Remove unnecessary getObjectMetadata() from getS3ObjectAndMetadata()
xinglin Nov 10, 2017
e6c1bbb
Remove one unused import to pass style check.
xinglin Nov 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Remove unnecessary getObjectMetadata() from getS3ObjectAndMetadata() 
ObjectMetadata is available from S3Object.getObjectMetadata().
We don't need to send a separate GetObjectMetadata Request (an
HTTP HEAD request), in order to get the metadata. This will double
performance for read workloads.

Signed-off-by: Xing Lin <Xing.Lin@netapp.com>
  • Loading branch information
@xinglin
xinglin committed Nov 10, 2017
commit 8cae4203d3a97c1f5ee4161b6b1700a3b1af8581
22 changes: 8 additions & 14 deletions s3/src/main/java/com/yahoo/ycsb/db/S3Client.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,11 +350,11 @@ protected Status writeToStorage(String bucket, String key,
totalSize = sizeArray*fieldCount;
} else {
try {
Map.Entry<S3Object, ObjectMetadata> objectAndMetadata = getS3ObjectAndMetadata(bucket, key, ssecLocal);
int sizeOfFile = (int)objectAndMetadata.getValue().getContentLength();
S3Object object = getS3ObjectAndMetadata(bucket, key, ssecLocal);
int sizeOfFile = (int)object.getObjectMetadata().getContentLength();
fieldCount = sizeOfFile/sizeArray;
totalSize = sizeOfFile;
objectAndMetadata.getKey().close();
object.close();
} catch (Exception e){
System.err.println("Not possible to get the object :"+key);
e.printStackTrace();
Expand Down Expand Up @@ -425,12 +425,12 @@ protected Status writeToStorage(String bucket, String key,
protected Status readFromStorage(String bucket, String key,
Map<String, ByteIterator> result, SSECustomerKey ssecLocal) {
try {
Map.Entry<S3Object, ObjectMetadata> objectAndMetadata = getS3ObjectAndMetadata(bucket, key, ssecLocal);
InputStream objectData = objectAndMetadata.getKey().getObjectContent(); //consuming the stream
S3Object object = getS3ObjectAndMetadata(bucket, key, ssecLocal);
InputStream objectData = object.getObjectContent(); //consuming the stream
// writing the stream to bytes and to results
result.put(key, new ByteArrayByteIterator(IOUtils.toByteArray(objectData)));
objectData.close();
objectAndMetadata.getKey().close();
object.close();
} catch (Exception e){
System.err.println("Not possible to get the object "+key);
e.printStackTrace();
Expand All @@ -440,23 +440,17 @@ protected Status readFromStorage(String bucket, String key,
return Status.OK;
}

private Map.Entry<S3Object, ObjectMetadata> getS3ObjectAndMetadata(String bucket,
private S3Object getS3ObjectAndMetadata(String bucket,
String key, SSECustomerKey ssecLocal) {
GetObjectRequest getObjectRequest;
GetObjectMetadataRequest getObjectMetadataRequest;
if (ssecLocal != null) {
getObjectRequest = new GetObjectRequest(bucket,
key).withSSECustomerKey(ssecLocal);
getObjectMetadataRequest = new GetObjectMetadataRequest(bucket,
key).withSSECustomerKey(ssecLocal);
} else {
getObjectRequest = new GetObjectRequest(bucket, key);
getObjectMetadataRequest = new GetObjectMetadataRequest(bucket,
key);
}

return new AbstractMap.SimpleEntry<>(s3Client.getObject(getObjectRequest),
s3Client.getObjectMetadata(getObjectMetadataRequest));
return s3Client.getObject(getObjectRequest);
}

/**
Expand Down