通过S3定制软件协议实现通用的文件存定制软件储服务中间件
引言
定制软件在日常开发文件上传相关服务时，定制软件通常都会选择腾讯云,阿里云,定制软件七牛云等提供的oss定制软件服务作为文件存储系统,定制软件如果需要自行搭建文件存储系统，定制软件通常则会采用等开源项目。
定制软件但是大家有没有考虑过，定制软件不同的厂商或者提供的客户端sdk定制软件都是不同的，如果项目开发过程中，需要切换底层文件系统，那么通常情况下意味着，我们需要完全替换掉相关文件上传代码，如果微服务项目，则需要替换掉所有使用到文件上传sdk微服务的代码，这显然会带来巨大的工作量。
为了解决上面这个问题，我们有如下两个思路:
- 项目中针对文件上传写出一个单独的抽象层接口，底层不同文件存储系统，提供对应的实现即可:
这个思路很容易想到，利用门面模型向调用方屏蔽底层实现,但是其实这里还有更加简洁的实现方式。
- 基本所有云服务厂商提供的oss服务和开源的oss项目都遵循了S3协议，是Simple Storage Service的缩写，即简单存储服务，因此其实我们这里利用这一点，写出一个通用的文件中间件，利用该中间件后，我们写的客户端api就对任何实现了S3协议的oss服务进行访问。
使用演示
这里我们以Minio作为演示案例，不清楚minio的可以查看minio官方文档学习一下，下面我们先用docker方式安装一下minio:
安装minio
docker pull minio/miniodocker run --name minio \-p 9000:9000 \-p 9090:9090 \-d --restart=always \-e "MINIO_ROOT_USER=admin" \-e "MINIO_ROOT_PASSWORD=admin123" \-v /usr/local/minio/data:/data \-v /usr/local/minio/config:/root/.minio \minio/minio server /data \--console-address '0.0.0.0:9090'- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
注意，这里要单独设置console的端口，不然会报错，且无法访问
这种安装方式 MinIO 自定义 Access 和 Secret 密钥要覆盖 MinIO 的自动生成的密钥
登录客户端（浏览器）：注意—>此处的端口，是你设置的console的端口：9090
此处的用户名密码为启动服务时，设置的用户名密码：admin admin123。
minio基本bucket操作不再详述，和普通的oss服务一样。
构建Starter
- gitee仓库地址
- 模块结构
- pom配置
<dependencies> <dependency> <groupId>com.amazonaws</groupId> <artifactId>aws-java-sdk-s3</artifactId> <version>1.12.267</version> </dependency> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.18.24</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-autoconfigure</artifactId> <version>2.3.12.RELEASE</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-configuration-processor</artifactId> <version>2.7.3</version> <optional>true</optional> </dependency></dependencies>- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 构建通用的遵循S3协议的oss服务接口
package com.oss.client;import com.amazonaws.services.s3.AmazonS3;import com.amazonaws.services.s3.model.PutObjectResult;import com.amazonaws.services.s3.model.S3Object;import java.io.IOException;import java.io.InputStream;/** * Oss 基础操作 * 想要更复杂操作可以直接获取AmazonS3，通过AmazonS3 来进行复杂的操作 * https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/examples-s3-buckets.html */public interface OssClient{ /** * 创建bucket * @param bucketName */ void createBucket(String bucketName); /** * 获取url * @param bucketName * @param objectName * @return */ String getObjectURL(String bucketName, String objectName); /** * 获取存储对象信息 * @param bucketName * @param objectName * @return */ S3Object getObjectInfo(String bucketName, String objectName); /** * 上传文件 * @param bucketName * @param objectName * @param stream * @param size * @param contextType * @return * @throws IOException */ PutObjectResult putObject(String bucketName, String objectName, InputStream stream, long size, String contextType) throws IOException; default PutObjectResult putObject(String bucketName, String objectName, InputStream stream) throws IOException{ return putObject(bucketName,objectName,stream, stream.available(), "application/octet-stream"); } AmazonS3 getS3Client();}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 实现类
package com.oss.client;import com.amazonaws.services.s3.AmazonS3;import com.amazonaws.services.s3.model.ObjectMetadata;import com.amazonaws.services.s3.model.PutObjectRequest;import com.amazonaws.services.s3.model.PutObjectResult;import com.amazonaws.services.s3.model.S3Object;import lombok.RequiredArgsConstructor;import java.io.IOException;import java.io.InputStream;import java.net.URL;/** * s3 是一个协议 * S3是Simple Storage Service的缩写，即简单存储服务 * @author zdh */@RequiredArgsConstructorpublic class S3OssClient implements OssClient { private final AmazonS3 amazonS3; @Override public void createBucket(String bucketName) { if (!amazonS3.doesBucketExistV2(bucketName)) { amazonS3.createBucket((bucketName)); } } @Override public String getObjectURL(String bucketName, String objectName) { URL url = amazonS3.getUrl(bucketName, objectName); return url.toString(); } @Override public S3Object getObjectInfo(String bucketName, String objectName) { return amazonS3.getObject(bucketName, objectName); } @Override public PutObjectResult putObject(String bucketName, String objectName, InputStream stream, long size, String contextType) throws IOException { ObjectMetadata objectMetadata = new ObjectMetadata(); objectMetadata.setContentLength(size); objectMetadata.setContentType(contextType); PutObjectRequest putObjectRequest = new PutObjectRequest(bucketName, objectName, stream, objectMetadata); putObjectRequest.getRequestClientOptions().setReadLimit(Long.valueOf(size).intValue() + 1); return amazonS3.putObject(putObjectRequest); } @Override public AmazonS3 getS3Client() { return amazonS3; }}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 对应的配置类
package com.oss.config;import lombok.Data;import org.springframework.boot.context.properties.ConfigurationProperties;@ConfigurationProperties(prefix = "oss")@Datapublic class OssProperties { private boolean enable = true; private String accessKey; private String accessSecret; /** * endpoint 配置格式为 * 通过外网访问OSS服务时，以URL的形式表示访问的OSS资源，详情请参见OSS访问域名使用规则。OSS的URL结构为[$Schema]://[$Bucket].[$Endpoint]/[$Object] * 。例如，您的Region为华东1（杭州），Bucket名称为examplebucket，Object访问路径为destfolder/example.txt， * 则外网访问地址为https://examplebucket.oss-cn-hangzhou.aliyuncs.com/destfolder/example.txt * https://help.aliyun.com/document_detail/375241.html */ private String endpoint; /** * refer com.amazonaws.regions.Regions; * 阿里云region 对应表 * https://help.aliyun.com/document_detail/31837.htm?spm=a2c4g.11186623.0.0.695178eb0nD6jp */ private String region; private boolean pathStyleAccess = true;- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 自动配置类
package com.oss;import com.amazonaws.auth.AWSCredentials;import com.amazonaws.auth.AWSCredentialsProvider;import com.amazonaws.auth.AWSStaticCredentialsProvider;import com.amazonaws.auth.BasicAWSCredentials;import com.amazonaws.client.builder.AwsClientBuilder;import com.amazonaws.services.s3.AmazonS3;import com.amazonaws.services.s3.AmazonS3Client;import com.oss.client.OssClient;import com.oss.client.S3OssClient;import com.oss.config.OssProperties;import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;import org.springframework.boot.context.properties.EnableConfigurationProperties;import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import java.util.Objects;import java.util.stream.Stream;/** * OSS服务自动配置类 * @author zdh */@Configuration@EnableConfigurationProperties(OssProperties.class)public class OssAutoConfiguration { @Bean @ConditionalOnMissingBean(S3OssClient.class) public OssClient ossClient(AmazonS3 amazonS3) { return new S3OssClient(amazonS3); } /** * 参考文档 * https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/credentials.html * 区域选择这块 * https://docs.aws.amazon.com/zh_cn/sdk-for-java/v1/developer-guide/java-dg-region-selection.html * @param ossProperties * @return */ @Bean @ConditionalOnMissingBean(AmazonS3.class) @ConditionalOnProperty(prefix = "oss", name = "enable", havingValue = "true") public AmazonS3 amazonS3(OssProperties ossProperties) { long nullSize = Stream.<String>builder() .add(ossProperties.getEndpoint()) .add(ossProperties.getAccessSecret()) .add(ossProperties.getAccessKey()) .build() .filter(s -> Objects.isNull(s)) .count(); if (nullSize > 0) { throw new RuntimeException("oss 配置错误,请检查"); } AWSCredentials awsCredentials = new BasicAWSCredentials(ossProperties.getAccessKey(), ossProperties.getAccessSecret()); AWSCredentialsProvider awsCredentialsProvider = new AWSStaticCredentialsProvider(awsCredentials); return AmazonS3Client.builder() .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(ossProperties.getEndpoint(), ossProperties.getRegion())) .withCredentials(awsCredentialsProvider) .disableChunkedEncoding() .withPathStyleAccessEnabled(ossProperties.isPathStyleAccess()) .build(); }}- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
- 68
- 69
- 70
- 71
测试
- 将starter进行打包安装到本地仓库
- 创建一个springboot项目，并在该工程导入该starter进行单元测试
打包的时候，可以将starter项目里面的lombok依赖去掉
- 添加配置属性
#对于minio来说,配置如下oss: endpoint: http://minio服务器所在ip:9000 access-key: admin access-secret: admin123 enable: true- 1
- 2
- 3
- 4
- 5
- 6
- 编码测试
@Test public void testCreateBucket(){ //hutool提供的spring快捷工具 OssClient ossClient = SpringUtil.getBean(OssClient.class); ossClient.createBucket("sale"); }- 1
- 2
- 3
- 4
- 5
- 6
@Test public void testUploadImg() throws IOException { //hutool提供的spring快捷工具 OssClient ossClient = SpringUtil.getBean(OssClient.class); ossClient.putObject("sale","dhy.img",new FileInputStream("xpy.png")); }- 1
- 2
- 3
- 4
- 5
- 6
@Test public void testgetImgUrl() throws IOException { //hutool提供的spring快捷工具 OssClient ossClient = SpringUtil.getBean(OssClient.class); String objectURL = ossClient.getObjectURL("sale", "dhy.img"); System.out.println(objectURL); }- 1
- 2
- 3
- 4
- 5
- 6
- 7