使用 Helm 安装 metersphere 2.10.6-lts 版本

应公司测试人员要求，部署最新版 metersphere 以供日常工作需求。

基于 metersphere 官方部署文档： https://metersphere.io/docs/v2.x/installation/kubernetes_installation/

找到 Giuhub 仓库： https://github.com/metersphere/helm-chart/tree/metersphere-2.10.6-lts

复制 values.yaml 文件到本地局部修改后直接部署

helm upgrade --install metersphere --create-namespace --namespace metersphere metersphere/metersphere -f metersphere-2.10.6-lts-offical.yaml

但是部署后会发现一系列如下问题，下面逐个记录是如何解决的

pvc 挂载失败
mysql 容器启动失败
system-setting 自服务启动失败

Q1: pvc 挂载失败，导致pod一直处于pending的状态

原因：
因为我使用的是azure cloud，而helm源码中定义的data和logs两种pvc的Access Modes都是ReadWriteMany类型，而我所部署的ASK中默认 storageClass default 不支持 ReadWriteMany类型

解决方式:

修改成引用支持更多Access Modes的 storageClass: azurefile

Q2: mysql 容器启动失败，一直报 mysql 库下的各个表不存在，如: Table 'mysql.plugin' doesn't exist

初始怀疑是镜像的问题，他家的镜像是 registry.cn-qingdao.aliyuncs.com/metersphere/mysql:8.0.33

随着后面问题的逐渐深入，发现可能真正原因还是和 pvc 有关

Q3: metersphere-system-setting 服务自启动失败

Caused by: org.flywaydb.core.internal.command.DbMigrate$FlywayMigrateException: Schema `metersphere` contains a failed migration to version 68 !
	at org.flywaydb.core.internal.command.DbMigrate.migrateGroup(DbMigrate.java:220)
	at org.flywaydb.core.internal.command.DbMigrate.lambda$migrateAll$0(DbMigrate.java:141)
	at org.flywaydb.database.mysql.MySQLNamedLockTemplate.execute(MySQLNamedLockTemplate.java:58)
	at org.flywaydb.database.mysql.MySQLConnection.lock(MySQLConnection.java:152)
	at org.flywaydb.core.internal.schemahistory.JdbcTableSchemaHistory.lock(JdbcTableSchemaHistory.java:144)
	at org.flywaydb.core.internal.command.DbMigrate.migrateAll(DbMigrate.java:141)
	at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:98)
	at org.flywaydb.core.Flyway.lambda$migrate$0(Flyway.java:188)
	at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:196)
	at org.flywaydb.core.Flyway.migrate(Flyway.java:140)
	at org.springframework.boot.autoconfigure.flyway.FlywayMigrationInitializer.afterPropertiesSet(FlywayMigrationInitializer.java:66)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1817)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1766)
	... 32 common frames omitted

这个问题发生在当我改为使用外部的 MySQL 后，因为Q2的问题一直没解决，所以想着换用外部的MySQL可以绕过MySQL 不可用的问题，但是当使用外部的数据库后，他的服务中又开始报 migration 数据相关的错，但是按照官方提供的修复 SQL 脚本也没修复成功。

https://github.com/metersphere/metersphere/blob/v2.10.6-lts/system-setting/backend/src/main/resources/db/migration/V68__modify_api_test_case.sql

最终，局部小改官方的默认 values.yaml -- 继续使用default pvc,但把 data 和 logs 类型的 pvc accessModes 改成 ReadWriteOnce 后重新部署问题得到解决

helm upgrade --install metersphere --create-namespace --namespace metersphere metersphere/metersphere -f metersphere-2.10.6-lts-offical.yaml

下面附上修改后的 values.yaml 文件

ingress:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-For "$remote_addr, $server_addr";
    nginx.ingress.kubernetes.io/proxy-body-size: 500m
    nginx.ingress.kubernetes.io/server-snippet: |
      client_header_buffer_size   1024k;
      large_client_header_buffers 8 2048k;
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  enabled: true
  host: t-metersphere.test.dev

common:
  imagePrefix: "registry.cn-qingdao.aliyuncs.com/metersphere/"
  imagePullSecrets: nil
  storageClass: default
  imageTag: v2.10.6-lts
  imagePullPolicy: Always
  properties: |-
    ## DATABASE
    spring.datasource.url=jdbc:mysql://{{.Values.mysql.host}}:{{.Values.mysql.port}}/metersphere?autoReconnect=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowPublicKeyRetrieval=true
    spring.datasource.username={{.Values.mysql.username}}
    spring.datasource.password={{.Values.mysql.password}}

    ## redis
    spring.session.store-type=redis
    spring.redis.redisson.file=file:/opt/metersphere/conf/redisson.yml

    ## KAFKA
    kafka.partitions=1
    kafka.replicas=1
    kafka.topic={{.Values.kafka.metricTopic}}
    kafka.bootstrap-servers={{.Values.kafka.host}}.{{.Release.Namespace}}:{{.Values.kafka.port}}
    kafka.log.topic={{.Values.kafka.logTopic}}
    kafka.test.topic={{.Values.kafka.testTopic}}
    kafka.report.topic={{.Values.kafka.reportTopic}}
    tcp.mock.port=10000

    ## minio
    minio.endpoint=http://{{.Values.minio.host}}:{{.Values.minio.port}}
    minio.access-key={{.Values.minio.username}}
    minio.secret-key={{.Values.minio.password}}

    ## JMETER
    jmeter.image={{ .Values.common.imagePrefix }}{{.Values.jmeter.image}}:{{.Values.jmeter.imageTag}}
    jmeter.pod.threads.limit=500

    ## K8S
    k8s.node-controller-image={{ .Values.common.imagePrefix }}{{.Values.nodeController.image}}:{{.Values.common.imageTag}}

    ## spring cloud
    eureka.client.service-url.defaultZone=http://{{.Values.eureka.host}}:{{.Values.eureka.port}}/eureka/

    logger.sql.level=info

apiTest:
  enabled: true
  image: api-test
  replicas: 1

performanceTest:
  enabled: true
  image: performance-test
  replicas: 1

systemSetting:
  enabled: true
  image: system-setting
  replicas: 1

projectManagement:
  enabled: true
  image: project-management
  replicas: 1

reportStat:
  enabled: true
  image: report-stat
  replicas: 1

testTrack:
  enabled: true
  image: test-track
  replicas: 1

gateway:
  enabled: true
  image: gateway
  replicas: 1

eureka:
  enabled: true
  image: eureka
  host: metersphere-eureka
  port: 8761
  replicas: 1

dataStreaming:
  enabled: true
  image: data-streaming
  replicas: 1
  properties: |-
    ## DATABASE
    spring.datasource.url=jdbc:mysql://{{.Values.mysql.host}}:{{.Values.mysql.port}}/metersphere?autoReconnect=false&useUnicode=true&characterEncoding=UTF-8&characterSetResults=UTF-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowPublicKeyRetrieval=true
    spring.datasource.username={{.Values.mysql.username}}
    spring.datasource.password={{.Values.mysql.password}}

    ## KAFKA
    kafka.partitions=1
    kafka.replicas=1
    kafka.topic={{.Values.kafka.metricTopic}}
    kafka.bootstrap-servers={{.Values.kafka.host}}:{{.Values.kafka.port}}
    kafka.log.topic={{.Values.kafka.logTopic}}
    kafka.test.topic={{.Values.kafka.testTopic}}
    kafka.report.topic={{.Values.kafka.reportTopic}}
    jmeter.report.granularity=5000

    ## minio
    minio.endpoint=http://{{.Values.minio.host}}:{{.Values.minio.port}}
    minio.access-key={{.Values.minio.username}}
    minio.secret-key={{.Values.minio.password}}

uiTest:
  enabled: false
  image: ui-test
  replicas: 1

workstation:
  enabled: true
  image: workstation
  replicas: 1

nodeController:
  enabled: true
  image: node-controller
  replicas: 1
  properties: |-
    ## TBD

redisson:
  yml: |-
    singleServerConfig:
      password: {{.Values.redis.password}}
      address: "redis://{{.Values.redis.host}}:{{.Values.redis.port}}"
      database: {{.Values.redis.database}}

jmeter:
  image: jmeter-master
  imageTag: 5.5-ms6-jdk17

logPersistence:
  enabled: true
  accessModes: ReadWriteOnce
  size: 10Gi

dataPersistence:
  enabled: true
  accessModes: ReadWriteOnce
  size: 10Gi

mysql:
  enabled: true
  image: mysql
  imageTag: "8.0.33"
  host: metersphere-mysql
  port: 3306
  username: root
  password: Password123@mysql#Test
  persistence:
    enabled: true
    accessModes: ReadWriteOnce
    size: 20Gi
minio:
  enabled: true
  image: minio
  imageTag: "RELEASE.2023-04-13T03-08-07Z"
  username: admin
  password: Password123@minio#Test
  host: metersphere-minio
  port: 9000
  persistence:
    enabled: true
    accessModes: ReadWriteOnce
    size: 20Gi
redis:
  enabled: true
  image: redis
  imageTag: "6.2.6"
  password: Password123@redis#Test
  host: metersphere-redis
  port: 6379
  database: 1
  persistence:
    enabled: true
    accessModes: ReadWriteOnce
    size: 10Gi
kafka:
  enabled: true
  fullnameOverride: metersphere-kafka
  host: metersphere-kafka
  port: 9092
  metricTopic: JMETER_METRICS
  logTopic: JMETER_LOGS
  testTopic: LOAD_TESTS
  reportTopic: JMETER_REPORTS
  persistence:
    enabled: false
  logPersistence:
    enabled: false
  logFlushIntervalMessages: _10000
  logFlushIntervalMs: 1000
  logRetentionBytes: _1073741824
  logRetentionCheckIntervalMs: 300000
  logRetentionHours: 168
  logSegmentBytes: _1073741824
  maxMessageBytes: _1000012
  livenessProbe:
    initialDelaySeconds: 20
    periodSeconds: 15
    timeoutSeconds: 15
  readinessProbe:
    initialDelaySeconds: 20
    periodSeconds: 15
    timeoutSeconds: 15
  externalAccess:
    enabled: true
    service:
      type: NodePort
      useHostIPs: true
    autoDiscovery:
      enabled: true
  serviceAccount:
    create: true
  rbac:
    create: true
  extraEnvVars:
    - name: FORMAT_MESSAGES_PATTERN_DISABLE_LOOKUPS
      value: "true"
  kraft:
    enabled: true
  zookeeper:
    enabled: false